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P  A  system  for  conducting  Training  Effectiveness  Evaluation  (TEE)  on  Army  Air  Defense  Training  is  described 
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process  evaluation;  (4)  tninirdg  materials  for  TEE  analysts;  and  (5)  a  set  of  worksheets  keyed  to  the  Guidelines. 
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The  Guidelines  for  Conducting  e  Training  Effectiveness  Evaluation  (TEE), 
Voluaes  1-3  were  developed  in  response  to  Any  Training  Study  HRN  79-269, 
entitled  Methodology  for  Training  Effectiveness  Analysis  (TEA). 

The  three  voluaes  of  the  Guidelines  explain  how  to  use  the  TEE  methodol¬ 
ogy  to  evaluate  a  course  of  instruction  and  to  formulate  revisions  for  cor¬ 
recting  training  deficiencies,  in  ongoing  courses  and  in  training  for 
operational  tests  (OTvs)  of  new  or  improved  equipment.  Both  product  analyses 
of  training  materials  and  process  analyses  of  training  methods  are  covered  by 
the  Guidelines.  Volumes  I  and  III  are  directed  toward  the  TEE  analyst  and 
associate  analysts  and  explain  how  to  conduct  a  TEE  and  how  to  remedy  defi¬ 
ciencies,  respectively.  Volume  II  addresses  the  needs  of  the  TEE  data  col¬ 
lector. 
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GLOSSARY 


CLASSROOM 

INSTRUCTION: 

COMMON  ERRORS: 

COMPONENT: 

CONCEPT: 

CONDITIONS: 

CONTENT  TYPE: 

CRITICAL 

CHARACTERISTICS: 

DEMONSTRATION: 

ENTRY  SKILLS: 

ENTRY  TEST: 

EXAMPLE: 


A  lecture  or  discussion  involving  a  group  of  students  and  an 
instructor.  Statements,  statement  help,  examples,  and 
example  help  are  the  most  commonly  used  presentation 
components  in  the  classroom;  however,  practice,  feedback, 
and  feedback  help  could  possible  be  employed  for  some  non¬ 
equipment  related  tasks  or  for  remember  level  objectives. 

Errors  commonly  made  while  performing  a  task  on  the  job. 

See  PRESENTATION  COMPONENT. 

A  set  of  specific  objects,  symbols,  or  events,  which  are 
grouped  together  on  the  basis  of  shared  (critical) 
characteristics  and  which  can  be  referenced  by  a  particular 
name  or  symbol  (see  CRITICAL  CHARACTERISTICS). 

The  "givens”  of  a  performance.  They  describe  the 
circumstances  under  which  the  task  (objective,  or  team 
function)  is  performed.  Conditions  are  those  things  which 
are  specifically  given  or  denied  to  the  student  at  the  time 
he  performs  the  action  specified  in  the  objective. 

• 

A  category  of  subject  matter  which  requires  the  same 
general  kind  of  learning  strategies.  Content  types  include 
fact,  concept,  procedure,  rule,  and  principle  (see  entries  for 
each  content  type). 

A  set  of  features  which  define  a  concept.  Any  object, 
symbol,  or  event  which  has  those  features  is  included  in  the 
concept  class.  Anything  which  lacks  even  one  of  the 
features  in  the  set  is  not  a  member  of  the  concept  class 
(i.e.,  a  non-exampie). 

An  actual  performance  of  a  procedure  for  purposes  of 
showing  the  learner  how  it  is  done.  Demonstrations  are 
usually  best  when  live  or  conducted  with  com munclat ions 
media  (e.g.,  film,  videotape,  tape/sllde),  but  may  be  written 
when  necessary. 

The  skills  and  knowledge  required  of  students  entering  a 
course  necessary  for  them  to  succeed  in  the  course. 

A  test  covering  course  entry  skills  given  before  the  course 
begins.  Used  to  restrict  course  enrollment  to  students 
possessing  the  entry  skills. 

A  presentation  com|>onent  in  which  the  student  is  told  or 
shown  how  a  statement  of  a  concept,  procedure,  rule,  or 
principle  applies  in  a  specific  case.  Also  the  specific  case 
itself  which  possesses  the  critical  characteristics  of  the 
concept  class. 


GLOSSARY 


EXAMPLE  HELP: 


FACT: 


FEEDBACK: 


FEEDBACK  HELP: 

FORMAT: 

HELP: 

HELP  SESSION: 


HIGHER-ORDER 

RULE: 

INDICATOR 

BEHAVIORS: 


INDIVIDUAL  STUDY: 


INTEGRATED 
PRACTICE  OR  TEST: 


JOB  PERFORMANCE 
AID  (3PA): 

LEARNING 
OBJECTIVE  (LO): 


Additional  information  given  following  the  presentation  of 
examples  to  aid  students  in  understanding  them. 

A  simple  piece  of  information  to  be  remembered,  such  as 
an  object,  event,  name,  part,  function,  location,  or  date, 
alone  or  in  combination  with  others. 

Information  given  to  a  student  as  to  the  correctness  of  a 
practice  or  test  performance. 

Explanation  of  why  an  answer  to  a  test  or  practice  item 
was  wrong. 

See  TEST  FORMAT. 

Additional  information  about  a  statement,  example,  or 
feedback  to  aid  the  students  in  understanding  it. 

Instruction  of  any  of  a  number  of  the  types  held  in  addition 
to  the  regular  course  instruction  to  aid  students  who  need 
additional  help.  Statement  or  example  help  as  well  as 
practice  and  feedback  would  be  the  typical  presentation 
components  employed. 

A  rule  made  up  of  other  concepts,  rules,  and/or  principles 
which  is  constructed  and  applied  when  using  a  principle. 

Actions  specified  in  an  objective  which  are  performed  to 
indicate  that  the  student  has  acquired  the  required  skill  or 
knowledge. 

A  situation  in  which  students  are  allowed  to  study  on  their 
own.  It  could  follow  classroom  instruction  or  it  could 
occur  in  individual  instructional  courses  where  students 
typically  proceed  at  their  own  pace. 

Practice  or  testing  of  two  or  more  tasks  in  conjunction 
with  each  other  at  a  high  level  of  realism  or  in  an 
operational  setting.  This  may  be  done  by  an  individual,  a 
group  performing  indivdiual  tasks,  or  a  team  performing 
team  functions. 

Any  written  or  pictorial  device  showing  the  steps  in  a 
procedure  or  rule  or  giving  a  decision  rule  for  concept 
classification.  Used  as  an  aid  during  job  performance. 

A  subunit  of  terminal  learning  objective  consisting  of  a 
precise  description  of  what  is  to  be  learned  in  terms  of  the 
expected  student  performance  under  specified  conditions  to 
accepted  standards.  These  learning  objectives  identify  the 
mental  skills,  information,  attitudes,  or  physical  skills  that 
are  required  to  perform  the  terminal  learning  objective. 

(See  TERMINAL  LEARNING  OBJECTIVE.) 
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GLOSSARY 


LESSON: 

NON-CRITICAL 

CHARACTERISTICS: 

NON-EXAMPLE: 

OBJECTIVE: 

ORAL  TEST: 

PERFORMANCE 

TEST: 

PRACTICE: 

PRACTICE 

REMEMBERING: 

PRACTICE  USING: 

PRESENTATION 

COMPONENT: 

PRETEST: 

PRINCIPLE: 


PROCEDURE: 


A  unit  of  instruction  teaching  one  or  more  tasks  or  parts 
of  tasks  usually  encompassing  a  short  series  of  training 
events. 

Features  which  do  not  define  a  concept,  but  may  be 
possessed  by  one  or  more  of  the  members  of  the  concept 
class  (see  CONCEPT,  CRITICAL  CHARACTERISTICS). 

An  object,  symbol,  or  event  which  does  not  contain  all  of 
the  critical  characteristics  of  a  concept  class,  usually  used 
in  contrast  to  an  example  during  instruction. 

A  precise  statement  of  a  behavior  to  be  learned  containing 
a  statement  of  the  required  action  along  with  conditions 
and  standards  of  performance. 

Assessment  of  how  well  students  can  perform  at  any  task 
level  using  spoken  responses  by  the  students. 

Assessment  of  how  well  students  can  perform  at  the  use 
task  levels  in  isolation  from  other  tasks, 

A  presentation  component  in  which  students  are.  given  the 
opportunity  to  remember  facts,  classify  examples  and  non¬ 
examples  of  concepts,  perform  procedures,  demonstrate 
rules,  or  apply  principles. 


Practice  at  the  remember  task  level. 

Practice  at  the  use-aided  or  use-unaided  task  levels. 

A  basic  unit  of  instruction;  either  a  statement,  example, 
practice,  or  feedback. 

A  test  given  at  the  beginning  of  a  course  on  course 
content;  used  to  exempt  students  from  parts  of  the  course 
they  have  already  mastered. 

A  general  rule  for  the  development  and  application  of  a 
"higher-order"  rule.  In  principle-using  situations  the 
behavior  involved  is  that  of  recalling  subordinate  concepts, 
rules,  or  principles;  combining  them  in  different  sets;  and 
trying  them  out  until  the  combination  (the  higher-order 
rule)  that  works  is  discovered. 

A  set  of  ordered  steps  designed  to  solve  a  specific  problem 
that  always  presents  itself  in  the  same  way. 
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GLOSSARY 


PROCESS 

EVALUATION: 

The  observation  of  the  process  of  training  as  it  occurs 
in  the  classroom  or  field  setting  and  the  analysis  of  the 
resulting  data. 

PRODUCT 

EVALUATION: 

The  analysis  of  printed  or  otherwise  mediated  training 
materials  to  ascertain  their  potential  effectiveness. 

RELIABILITY: 

See  TEST  RELIABILITY. 

REMEMBER 

TASK  LEVEL: 

A  dimension  of  task  performance  in  which  the  student 
is  required  to  remember  either  facts,  concept  definitions, 
steps  in  a  procedure  or  rule,  or  potential  causes/effects 
useful  in  principle  building. 

STANDARD: 

A  portion  of  a  task,  objective,  or  team  function  which 
specifies  how  well  the  action  must  be  performed. 

STATEMENT: 

A  presentation  component  in  which  the  student  is  given  a 
statement  of  a  fact,  a  concept  definition,  the  steps  of  a 
procedure  or  rule,  or  a  statement  of  a  principle. 

STATEMENT  HELP: 

Additional  information  given  following  the  presentation  of  a 
statement  to  aid  students  in  understanding  it. 

RULE: 

A  set  of  ordered  operations  or  steps  which,  when  applied, 
will  solve  a  particular  type  or  class  of  problems. 

SUBORDINATE 

SKILLS: 

Tasks  which  must  be  learned  prior  to  the  learning  of  other 
(superordinate)  tasks. 

SUPERORDINATE 

SKILLS: 

Tasks  in  a  superior  relationship  to  other  (subordinate)  tasks. 
Superordinate  tasks  cannot  be  learned  until  their  subordinate 
tasks  are  mastered. 

TASK: 

A  highly  specific  component  of  a  job  stated  in  terms  of  a 
verb  and  an  object  with  conditions  and  standards  of 
performance. 

TASK  ELEMENT: 

A  step  in  a  task  that  is  just  small  enough  that  it  requires 
no  explanation  for  a  non-expert  to  understand  what  it 
means. 

TASK  LEVEL: 

A  general  classification  of  task  or  objective  actions,  either 
remember,  use-aided,  or  use-unaided}  (see  the  respective 
task  levels). 

TEAM  FUNCTION: 

An  action  to  be  performed  by  a  team  or  crew  requiring 
multiple  tasks  to  be  performed  by  individuals  in  conjunction 
with  each  other. 

TEAM  PRACTICE: 

Practice  of  a  team  function  by  all  crew/team  members  in  a 
realistic  setting. 
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GLOSSARY 


TEAM  PRACTICE 
CONDITIONS: 


TEST  ADEQUACY: 


TEST  FORMAT: 


TEAM  PRACTICE 
FEEDBACK: 

TERMINAL  LEARNING 
OBJECTIVE  (TLO): 


TEST  RELIABILITY: 


TEST  VALIDITY: 


TRAINING  EVENT: 


USE-AIDED  TASK 
LEVEL: 

USE-CONCEPT, 
USE-PROCEDURE, 
OR  USE-PRINCIPLE: 

USE-UNAIDED 
TASK  LEVEL: 

VALIDITY: 

WRiTTEN  TEST: 


The  circumstances  under  which  a  team  function  is 
practiced  including  the  resources  available  to  team 
members. 

The  extent  to  which  a  test  conforms  to  factors  contributing 
to  validity  and  reliability. 

The  type  of  test  item  used  in  an  examination,  either 
true/false,  multiple  choice,  matching,  fill-in,  short  answer, 
listing,  or  performance. 

Feedback  given  to  a  team  following  practice  of  a  team 
function. 

A  specific  description  of  the  action,  with  conditions/ 
standards,  to  be  performed  after  training.  Derived  from 
job  performance  measures,  TLO's  are  to  be  attained  during 
training.  TLO’s  ar  roken  down  into  their  component 
parts  which  are  documented  as  learning  objectives  which 
may  be  further  divided  into  learning  steps. 

The  consistency  with  which  a  test  measures  the  amount  of 
student  achievement. 

Whether  test  items  or  tests  measure  what  they  are 
supposed  to  measure  as  defined  by  instructional  objectives. 

A  period  of  instruction  with  a  specific  purpose  usually 
involving  one  or  two  presentation  components. 

A  dimension  of  task  performance  in  which  the  student  uses 
a  job  performance  aid  in  performing  the  task. 

Terms  used  to  describe  objectives,  practice  items,  or  test 
items  at  the  use-aided  or  use-unaided  task  level  and  the 
designated  content  type. 

A  dimension  of  task  performance  in  which  the  student  has 
no  aids  to  task  performance  except  his  own  memory. 

See  TEST  VALIDITY. 

Assessment  of  how  well  students  can  perform  at  any  task 
level  using  paper  and  pencil  tests. 
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THE  TEE  SYSTEM 

The  TEE  system  consists  of  the  TEE  Evaluator's  Handbook:  Guidelines  for 
Conducting  a  Training  Effectiveness  Evaluation ,  a  Data  Collector’s  Manual,  job 
performance  aids  for  product  and  process  evaluation,  and  a  set  of  reproducible 
masters  for  worksheets  and  training  materials. 

The  TEE  Evaluator’s  Handbook  includes  the  following  major  components: 

a.  Guidelines  for  conducting  the  TEE  (Phases  A  through  F). 

b.  A  master  list  of  evaluation  questions. 

c.  Job  aids  for  conducting  product  and  process  evaluations. 

d.  Training  materials  for  TEE  analysts  and  associate  analysts. 

e.  A  set  of  worksheets  keyed  to  the  guidelines. 

Worksheets  are  used  with  11  out  of  the  18  TEE  tasks  in  conducting  an 
evaluation.  In  other  tasks,  actual  course  documentation  is  annotated  and  used  as  a 
worksheet.  Each  worksheet  is  for  the  purpose  of  either  planning  to  collect  data, 
collecting  data,  summarizing  data,  or  interpreting  data*  The  Guidelines  give  step-by- 
step  directions  and  examples  for  using  the  worksheets  to  conduct  an  evaluation. 

The  primary  components  of  the  Data  Collector’s  Manual  are: 

a.  The  master  list  of  evaluation  questions  edited  for  use  by  data 
collectors. 

b.  Training  materials  on  how  to  classify  task  level  and  to  recognize 
glossary  terms. 

The  job  performance  aid  for  use  in  product  evaluation  is  a  subset, of  the 
master  list  of  evaluation  questions.  It  contains  only  those  TEE  questions  appropriate 
to  product  evaluation  along  with  abbreviated  rating  guidance. 

The  job  performance  aid  for  use  In  process  evaluation  is  also  a  subset  of  the 
master  list.  It  contains  detailed  guidance  for  those  process  TEE  questions  which 
cannot  be  rated  without  directions  beyond  that  provided  in  the  questions  themselves. 


The  scope  of  the  TEE  Guidelines  extends  from  the  initial  request  to  perform 
an  evaluation  on  a  designated  Army  *  xirse  to  a  report  outlining  the  results  of  the 
investigation.  The  scope  of  a  TEE  project  includes  collecting  data  for  the  purposes 
of  identifying  failures  in  performance  of  tasks  by  the  trainees  (called  performance 
discrepancies  in  the  Guidelines)  and  failures  in  the  instructional  system  itself.  Data 
on  both  the  conduct  of  training  (derived  from  a  process  evaluation),  and  on  the 
training  materials  (derived  from  a  product  evauation)  can  be  included. 

Along  these  same  lines,  it  may  not  always  be  necessary  to  conduct  a  ’’full 
TEE”  using  every  task  and  step  outlined  in  Figures  1  and  2.  Some  of  the  data 
collection  steps  may  be  omitted  when  corresponding  information  is  already  known,  or 
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Figure  1  CONCEPTUAL  MODEL  OF  TEE  PROCEDURES 


Figure  2  FLOWCHART  OF  TEE  TASKS 


one  may  wish  to  concentrate  on  certain  aspects  of  the  training  which  are  known  to 
be  bad,  A  complete  TEE  may  not  be  within  the  resources  available. 

A  number  of  different  types  of  personnel  will  be  involved  in  the  TEE.  In 
addition  to  the  sponsor,  the  organization  that  decides  to  conduct  the  TEE  and 
utilizes  the  results,  and  the  training  organization  which  will  be  evaluated,  there  are 
three  possible  types  of  personnel  who  may  also  be  involved: 

a.  The  TEE  analyst  will  be  responsible  for  planning  and  conducting  the 
TEE,  analyzing  the  data,  and  writing  the  final  report  on  the  TEE.  He 
may  need  to  supervise  a  team  of  data  collectors.  He  should  have  some 
background  in  training  development  or  evaluation  and  should  preferably 
have  some  familiarity  with  Instructional  Systems  Development  (ISD,  see 
TRADOC  Pam  350-30).  Completion  of  an  ISD  Workshop  and/or  a 
Criterion  Referenced  Instruction  course  (or  their  equivalent)  is 
recommended. 

b.  An  associate  analyst  may  be  employed  to  assist  the  TEE  analyst  with 
some  or  all  of  his  responsibilities.  He  should  be  a  subject  matter 
expert  (5ME)  in  the  training  to  be  evaluated.  If  a  team  of  data 
collectors  (described  below)  is  employed,  one  of  his  primary 
responsibilities  should  be  to  supervise  much  of  the  data  collection 
activity,  acting  as  an  interface  between  the  TEE  analyst  and  the  data 
collectors.  In  that  event,  he  should  be  an  Army  officer, 
noncommissioned  officer,  or  enlisted  man  with  a  higher  rank  than  that 
of  the  data  collectors  selected  for  the  TEE.  He  need  not  have  a 
training  background,  although  that  would  be  desirable. 

c.  Data  collectors  should  be  employed  whenever  the  TEE  analyst  and 
associate  analyst  cannot  adequately  observe  the  training  alone.  This 
may  occur  when  there  are  several  classes  being  taught  at  the  same 
time.  Data  collectors  can  be  selected  from  the  pool  of  all  available 
personnel,  but  should  be  used  only  after  successful  completion  of 
training  on  TEE  observation  tasks.  It  may  also  be  desirable  to  have 
one  or  more  subject  matter  experts  among  them  as  an  additional 
resource  to  the  TEE  analyst. 

Another  characteristic  important  for  associate  analysts  and  data  collectors  is 
reading  ability.  Without  this  quality  they  will  be  difficult  to  train  and  will  not 
function  well  in  the  TEE.  The  associate  analyst  should  have  a  reading  grade  level 
of  at  least  10.  It  is  also  preferable  for  data  collectors  to  have  a  tenth  grade 
reading  ability.  However,  subject  matter  experts  (SMEs)  with  a  lower  reading  level 
may  be  paired  with  non-SMEs  who  have  a  tenth  grade  reading  ability.  In  no  case 
should  data  collectors  be  recruited  with  reading  grade  levels  lower  than  8. 
Approximate  reading  grade  levels  can  be  computed  from  AFQT  scores  by  using  the 
following  formula*:  Estimated  reading  grade  *  .075  (AFOT)  ♦  5.52  level. 

Especially  in  the  early  stages  of  the  YEE,  the  TEE  analyst  should  become 
well  acquainted  with  the  sponsor  (the  organization  requesting  the  TEE)  so  that  he 


^Stitch,  T.G.  (Ed.).  Reading  for  working:  a  functional  literacy  anthology. 
Alexandria,  VA:  Human  Resources  Research  Organization,  1975,  p.  80. 
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will  understand  what  is  expected  from  the  TEE  and  what  his  resources  will  be.  It 
is  also  important  that  the  channels  of  communication  be  opened,  putting  him  in 
direct  contact  with  the  training  organization  conducting  the  training  he  will 
evaluate. 

The  TEE  methodology  consists  of  six  phases: 


a. 

Phase  A: 

Plan  the  TEE. 

b. 

Phase  B: 

Conduct  product  evaluation. 

c. 

Phase  C: 

Plan  training  process  evaluation. 

d. 

Phase  D: 

Conduct  training  process  evaluation. 

e. 

Phase  E: 

Assess  trainee  performance. 

f. 

Phase  F: 

Document  the  TEE. 

Each  of  these  phases  is  described  in  the  following  scenario  of  a  typical  TEE. 
In  this  instance,  all  of  the  TEE  tasks  are  exercised,  as  would  be  required  for  an  OT 
training  evaluation. 

Phase  A:  Plan  the  TEE 

The  purpose  of  Phase  A  is  to  collect  as  much  background  and  logistical 
information  as  feasible  In  advance  of  observing  the  training  and  testing  that  will  be 
analyzed.  This  involves  assembly  of  documentation  and  plans  that  already  exist,  and 
an  initial  meeting  with  the  requestor/user  of  the  TEE.  It  also  involves  decisions 
regarding  the  TEEs  purpose  and  the  selection  of  appropriate  questions  to  ask  during 
the  TEE. 

Little  actual  judgment  of  the  training  and  materials  is  involved  In  Phase  A. 
The  effort  primarly  involves  getting  ready  to  perform  the  TEE.  However  it  docs 
involve  a  review  of  the  tasks  and  objectives  addressed  in  the  course. 

These  are  the  tasks  for  Phase  A: 

Task  Ah  Collect  background  Information  and  define  TEE  purpose.  In  this  task, 
the  TEE  analyst  requests  a  meeting  with  the  organizations  that  are 
most  knowledgeable  of  the  training  content  and  methods  of  the  course 
that  the  TEE  will  analyze.  He  also  requests  the  following  materials 
and  documentation  relevant  to  the  project: 

a.  Task  documentation  and/or  Job  Data  Worksheets. 

b.  ISD  or  other  documentation  relevant  to  the  training  events  that 
will  occur,  such  as  objectives,  hierarchies,  lesson  plans,  and 
practical  exercises. 

c.  Test  administration  directions  and/or  any  documentation 
concerning  the  methods  that  will  be  used  during  and  at  the  end 
of  the  course  to  evaluate  the  performance  of  the  trainees. 
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d.  Commander’s  manual  relevant  to  training. 

e.  Soldier’s  manual(s)  relevant  to  the  training. 

f.  Course  materials  to  be  used  by  the  trainees. 

g.  Testing/evaluation  instruments  to  be  used  in  the  course. 

h.  Training/testing  schedules. 

i.  Course  administration  policies. 

This  documentation  is  studied  prior  to  the  initial  meeting  with 
representatives  of  the  training  organization,  A  worksheet  entitled 
"Background  on  the  Project"  is  provided  to  organize  relevant  TEE 
background  information  and  for  use  as  an  agenda  for  the  Intial  meeting. 
Guidelines  for  completing  this  worksheet  are  included  in  the  TEE 
Evaluator’s  Handbook. 

Another  subtask  in  Task  A1  involves  determining  the  scope  of  the  TEE 
effort  in  light  of  the  TEE  purpose  and  corresponding  resource 
considerations.  The  TEE  analyst  makes  his  initial  determination  from 
an  informal  needs  assessment  and  from  reference  to  the  flowchart  of 
TEE  tasks  and  the  table  of  TEE  purposes  and  corresponding  task  options 
in  the  TEE  Evaluator's  Handbook.  A  preliminary  assessment  of  TEE 
purpose  (i.e.  operational  test,  Review,  or  Complaints),  corresponding 
TEE  tasks,  and  required  resources  is  made  to  define  the  scope  and  to 
guide  the  TEE  Analyst  In  making  recommendations  to  his  sponsor. 

The  TEE  Analyst  then  meets  with  his  sponsor  for  the  final 
determination  of  the  scope  of  the  TEE.  It  is  the  TEE  Analyst's 
responsibility  to  insure  effective  communication  so  that  a  complete 
understanding  of  the  effort  required  and  the  anticipated  results  is 
established  with  the  sponsor.  Once  this  understanding  is  achieved,  the 
TEE  Analyst  documents  all  commitments  in  writing. 

Task  A2i  Review  course  materials  and  document  task/TLO  actions*  The  analyst 
next  reviews  all  available  documentation  and  the  Information 
accumulated  via  the  Background  worksheet.  If  practical,  he  discusses 
the  key  training  tasks  to  be  taught  with  the  instructors  or  other 
subject  matter  experts. 

The  analyst  then  develops  an  accurate  list  of  tasks  or  objectives  and/or 
team  functions.  A  glossary,  self-instruction  and  practice  on  classifying 
content  types  and  task  levels,  and  "Guidelines  for  Task/TLO  Actions" 
are  provided  in  the  manual  as  supporting  reference  material  for  this 
effort. 

If  an  existing  list  o!  tasks  o;  objectives  is  unavailable  or  is  of  poor 
quality,  the  Guidelines  provide  for  options  of  extensively  revising  the 
existing  task  list  or  producing  a  new  one. 

Task  A3:  Select  tasks  to  be  evaluated  In  the  TEE,  in  this  task,  the  analyst 

decides  whether  or  not  he  can  evaluate  alt  of  the  tasks  in  the  course. 
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If  doing  so  is  beyond  existing  resources  or  is  impractical,  tasks  must  be 
selected  for  evaluation.  The  list  of  tasks  under  consideration  is 
iteratively  narrowed  down,  based  on  considerations  of  whether  each  task 
is:  already  known  by  the  majority  of  trainees;  known  to  have  had 
performance  problems  in  the  past;  difficult  to  learn;  performed 
frequently  on  the  job;  and  other  factors. 

Task  A4:  Document  conditions  and  standards*  The  purpose  of  this  task  is  to 

document  objectives  so  that  they  validly  reference  tasks  as  they  are 
performed  on  the  job.  Up  to  this  point,  the  analyst  has  examined  tasks 
in  terms  of  the  actions  the  trainee  will  be  expected  to  exhibit  after 
training.  Now  conditions  and  standards  for  tasks/terminal  learning 
objectives  (TLOs)  must  be  evaluated  or  added  if  not  already 
documented. 

"General  Guidance  on  Objectives"  and  "Guidelines  for  Evaluating 
Objectives"  are  included  in  the  TEE  manual  as  an  aid  to 
identifying/writing  complete  TLOs  which  correspond  to  tasks,  and 
learning  objectives  (LOs)  which  match  the  major  subtasks.  Criteria  are 
given  for  determining  whether  each  objective  (TLO  or  LO)  is  correctly 
stated,  classifiable  by  task  level  and  content  type,  and  appropriate. 
Guidance  is  also  provided  for  determining  whether  all  of  the  required 
LOs  for  a  TLO  are  present. 

Task  A5s  Specify  training  event  types  and  select  events  for  observation.  Using 
tnc  documentation  collected  in  previous  Phase  A  tasks,  the  ^analyst  now 
makes  a  list  of  the  types  of  training  events  that  are  employed  in  the 
course.  These  may  include: 

a.  Classroom  instruction. 

b.  Demonstration. 

c.  Practice. 

d.  Performance  test. 

e.  Written  test. 

f.  Oral  test. 

g.  Integrated  practice  or  test. 

h.  Individual  study. 

I.  Help  session. 

He  then  lists  the  training  events  associated  with  each  lesson  topic, 
annotates  this  list  with  tlx?  corresponding  task  numbers,  and  eliminates 
events  for  tasks  not  selected  for  evaluation. 
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Phase  8:  Conduct  Product  Evaluation 

The  purpose  of  Phase  B  is  to  evaluate  the  course  materials  and  note 
deficiencies  that  are  likely  to  cause  performance  problems  on  the  final  test.  The 
general  evaluation  strategy  is  to  check  to  see  that  test  items  are  adequate  and 
match  the  objectives,  end  to  check  the  planned  presentation  for  adequacy  and  its 
match  to  the  test  items. 

These  arc  the  tasks  for  Phase  B: 

Task  31:  Select  product  evaluation  questions.  In  this  task  the  master,  list  of 

evaluation  questions  (Appendix  A)  has  its  first  use.  A  list  of  the  "short 
forms"  of  all  master  list  questions,  along  with  their  rating  scales,  Is 
shown  in  Appendix  B.  The  master  list  is  divided  into  two  sections:  one 
for  tests,  and  one  for  the  presentation  of  the  Instruction.  In  Task  Bl, 
the  analyst  Identifies  those  questions  that  are  applicable  to  a  product 
evaluation  (conducted  using  the  course  materials),  and  to  the  training 
situation.  A  similar  process  is  conducted  for  process  evaluation  in  Task 
O. 

Task  82:  Evaluate  test  materials.  In  this  task,  the  TEE  analyst,  referring  to 

both  the  list  of  objectives  and  the  course  test(s),  identifies  the  test 
questions  that  test  each  objecitve.  He  then  classifies  each  test 
question  by  task  level  and  content  type,  eliminating  inapplicable  test 
items*  Finally  the  TEE  analyst  conducts  the  evaluation  of  applicable 
test  materials  by  asking  each  of  the  test-related  master  list  questions 
(not  previously  eliminated)  of  each  test  or  test  item,  as  appropriate.  A 
job  performance  aid  for  product  evaluation  is  provided  to  facilitate  this 
process. 

Task  B3:  Evaluate  presentation  materials.  The  first  step  In  this  task  is  to 

gather  the  necessary  course  documentation: 

a.  The  course  objectives. 

b.  All  lesson  materials  and  manuals  used  by  the  students. 

c.  Any  audio-visual  equipment  needed  to  hear  or  view  the  materials. 

d.  Lesson  and  course  administrative  directions. 

The  Product  Evaluation  JPA  and  the  course  outline  with  events  to  be 
evaluated  are  also  required  for  this  task.  The  analyst  conducts  the 
evaluation  of  presentation  materials  by  asking  each  question  in  the 
presentation  section  of  the  master  list,  of  each  lesson,  objective, 
presentation  component  (e.g.,  examples  or  practice),  or  the  course  as  a 
whole,  depending  upon  the  level  to  which  each  question  applies. 

Phase  C:  Plan  Training  Process  Evaluation. 

The  purpose  of  Phase  C  is  to  prepare  a  set  of  worksheets  for  recording 
training  process  observations  appropriate  to  the  TEE  setting  and  to  make  plans  to 
observe  specific  training  events.  General  worksheets  containing  all  possible  TEE 
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questions  for  different  event  types  are  included  in  an  appendix  to  the  TEE 
Evaluator’s  Handbook. 

No  actual  judgment  of  the  training  or  materials  is  involved  in  Phase  C.  The 
effort  primarily  involves  getting  ready  to  train  data  collectors  and  preparations  for 
observing  the  training  as  it  is  being  conducted  in  Phase  D. 

These  are  the  tasks  for  Phase  C: 

Task  Cl:  Prepare  process  evaluation  worksheets.  In  this  task  the  analyst  locates 

the  appropriate  TEE  worksheets  for  the  training  events  to  be  evaluated 
in  the  process  mode.  The  general  heading  of  each  worksheet  is 
modified  to  fit  the  analyst’s  particular  requirements.  TEE  questions 
which  are  inapplicable  to  the  events  under  consideration  are  eliminated 
from  the  worksheets.  The  analyst  also  has  the  option  of  creating  his 
own  process  evaluation  worksheets,  tailored  to  the  training  settings  to 
be  observed. 

The  steps  remaining  in  this  task  concarn  the  preparation  of  trainee  and 
instructor  reaction  instruments.  While  trainees  are  not  particularly 
qualified  to  make  subjective  judgments  about  the  quality  of  training, 
they  are  quite  capable  of  observing  what  happened  in  the  training  and 
stating  how  it  affected  either  the  way  they  learned  the  material  or 
their  motivation  for  learning  it.  Therefore,  trainee  reactions  which 
pinpoint  individual  learning  problems  or  points  in  the  instruction  that 
substantially  reduced  motivation  for  learning  are  appropriate  for  TEE 
data  collection. 

Task  C2:  Make  logistical  arrangements  to  conduct  the  training  process  TEE.  At 

tRis~poTnt7  the  tSE  analyst1  must  consider  how  many  training  events  to 
observe.  As  a  minimum,  the  following  events  must  be  observed  for 
each  task  selected  for  evaluation: 

a.  A  final  test  for  each  task  selected  for  evaluation. 

b.  At  least  50%  of  the  demonstrations  for  each  instructor  involved. 

c.  As  much  practice  as  Is  practical  for  each  task  selected. 

d.  As  much  classroom  instruction  as  possible. 

e.  As  many  of  the  other  training  event  types  as  possible. 

When  the  purpose  of  the  TEE  is  to  certify  training  prior  to  an 
operational  test  of  a  developing  weapons  system,  it  is  more  important 
to  observe  a  large  percentage  or  all  of  the  training  events  for  their 
entire  duration.  On  the  other  hand,  when  ongoing  training  is  observed 
that  appears  to  be  functioning  fairly  well  and  which  is  consistent  across 
instructors,  complete  observation  of  every  training  event  becomes  less 
important. 

After  deciding  which  events  to  observe,  the  analyst  prepares  an 
"observation  plan"  or  TEE  schedule,  and  communicates  his  plans  for  an 
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on-site  visit  to  the  personnel  in  charge  of  the  training  to  be  observed. 
At  this  time  the  analyst  obtains  and  reviews  all  training  material  and 
tests  for  the  events  to  be  evaluated,  if  he  has  not  already  done  so. 

Phase  Ds  Conduct  Training  Process  Evaluation. 

During  Phase  D,  the  training  and  testing  events  selected  in  Phase  A  are 
observed  directly.  The  primary  purpose  of  Phase  D  observations  is  to  gather 
information  that  will  be  useful  in  analyzing  the  results  of  the  training,  i.e.,  the 
performance  scores. 

If  the  trainees  are  unable  to  meet  performance  criteria  when  they  engage  in 
operational  tests,  the  information  collected  in  Phase  D  will  be  useful  in  determining 
the  causes  of  the  performance  deficiencies. 


If  the  trainees  are  able  to  perform  to  criteria,  the  observation  data  collected  in 
Phase  D  will  serve  as  a  record  of  the  training  and  testing  conditions  that  produced 
the  result.  A  record  of  success  is  often  as  helpful  to  the  designers  and  deliverers 
of  training  as  feedback  on  failures. 

Three  major  types  of  information  are  collected  during  this  phase: 

a.  Direct  observation  data. 

b.  Trainee  and  instructor  characteristics. 

c.  Trainee  and  instructor  reaction  data. 

These  are  the  tasks  for  Phase  D: 

Task  Db  Train  data  collectors.  In  this  task  the  TEE  analyst  familiarizes  data 
collectors  with  the  worksheets  they  will  use  in  observing  the  training 
and  trains  them  in  the  skills  necessary  to  do  so. 

The  analyst  first  insures  that  he  has  a  sufficient  number  of  data 
collectors,  if  indeed  the  course  has  sufficient  enrollment  to  require 
additional  personnel  to  collect  data. 

In  the  next  few  steps  the  analyst  gathers  the  appropriate  training 
materials,  gives  the  data  collectors  an  overview  of  the  TEE  process  and 
the  mechanics  of  what  they  will  be  doing,  conducts  training  on 
terminology  used  in  the  observation  worksheets  that  may  be  unfamiliar 
to  the  data  collectors,  and  administers  a  self-instructional  module  on 
how  to  classify  task  level,  a  skill  the  data  collectors  will  need  in 
answering  some  of  the  questions  on  the  worksheets. 

If  some  data  collectors  are  not  SME's,  the  analyst  must  insure  that  all 
of  them  are  familiar  with  the  tasks  to  be  evaluated.  He  may  have  one 
of  the  data  collectors  who  Is  a  SME  conduct  this  segment  of  the 
training. 
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In  the  last  few  steps,  the  analyst  gives  a  thorough  explanation  and 
discussion  of  the  questions  on  the  applicable  observation  worksheets, 
demonstrates  methods  for  interviewing  trainees  and  administering 
questionnaires,  and  gives  the  data  collectors  an  opportunity  to  practice 
making  ratings  on  a  segment  of  the  training. 

Task  D2:  Collect  data  on  training  and  testing  events*  In  this  task,  the  TEE 

analyst  sends  the  data  collectors  out  to  observe  the  training.  On  a 
daily  basis  he  prepares  assignments,  stating  where  to  go  and  what  to 
observe.  He  includes  the  appropriate  data  collection  worksheets 
prepared  in  Task  Cl  and  training  materials  needed  for  reference.  When 
the  data  collectors  return,  the  analyst  reviews  the  data  and  clarifies 
problem  areas  with  the  data  collectors.  He  decides  whether  data  are 
usable  or  not  and  whether  individual  data  collectors  can  indeed  collect 
good  data. 

When  a  series  of  training  events  has  been  observed  that  apply  to  an 
individual  task,  the  analyst  fills  out  a  special  worksheet  answering 
questions  that  are  broader  than  can  be  answered  by  observations  of  a 
single  training  event. 

Task  D3:  Collect  trainee  and  instructor  characteristics  data.  At  some  point 

during  the  TEE,  the  analyst  examines  available  personnel  records  or 
other  documentation  to  ascertain  course  entrance  requirements  and 
instructor  qualifications,  and  the  degree  to  which  students  and 
instructors  possess  them.  The  TEE  Handbook  provides  guidelines  for 
filling  out  worksheets  for  trainees  and  instructors. 


Phase  E:  Assess  Trainee  Performance. 

This  phase  has  four  major  purposes: 

a.  Make  assessments  of  how  well  trainees  can  perform  the  tasks  selected 
for  evaluation  upon  completion  of  training. 

b.  Judge  the  adequacy  of  the  above  assessment. 

c.  Summarize  observations  of  the  training  and  training  materials  as  an  aid 
to  identifying  performance  discrepancies  and  as  input  to  the  revision 
process. 

d.  Identify  tasks  on  which  performance  standards  are  not  met  — 
(performance  discrepancies). 

The  TEE  analyst  takes  ail  of  the  observational,  interview,  questionnaire,  and  test 
data  collected  during  the  TEE  and  synthesizes  two  primary  outputs:  a.  a  list  of 
tasks  for  which  the  standards  have  not  been  met,  and  b.  a  list  of  potential  problem 
areas  for  each  task.  Each  deficiency  will  have  been  rated  as  minor  or  serious  as  it 
would  impact  test  adequacy  or  student  performance. 
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Phase  E  Tasks  are: 

Task  El:  Collect  and  summarize  test  data.  In  this  task,  the  analyst  locates  all 

of  the  relevant  test  data  for  the  course  final  exam.  He  converts  to 
Go/No  Go  scoring  on  a  task  by  task  basis,  if  the  data  are  not  already 
in  that  form.  He  then  enters  the  data  on  a  worksheet  which  allows 
him  to  calculate  the  percentage  of  No  Gos  on  each  task  and  for  the 
entire  test.  The  same  process  can  be  accomplished  for  team  functions, 
if  crews  rather  than  individuals  are  scored. 

The  data  from  any  within-course,  entry,  or  pretests  are  summarized  in 
the  same  manner. 

Task  E2:  Summarize  product  and  process  evaluation  data.  For  process  evaluation 

data  which  do  not  stem  from  master  list  questions,  i.e.,  trainee  and 
instructor  reaction  instruments,  the  reactions  must  be  rated  on  a  three- 
point  scale.  Having  done  that,  the  analyst  averages  ratings  for  all  TEE 
data  for  questions  where  different  data  collectors  have  rated  the  same 
event.  The  averages  are  then  used  rather  than  the  raw  data. 

Reactions  by  trainees  and  instructors  are  labelled  as  pertaining  to 
product  or  process  evaluation. 

Testing  problems  are  then  summarized:  the  analyst  records  the 
question  number  and  rating  for  each  problem  noted  in  the  data  for 
each  task  or  learning  objective.  Product  and  process  ratings  are 
recorded  under  separate  headings,  which  are  segmented  further  into 
sections  for  questions  that  apply  to  each  test  item,  to  the  objective,  or 
to  the  test  as  a  whole.  The  analyst  then  uses  a  similar  method  to 
record  ratings  of  the  training  presentation.  When  serious  problems  a*  ? 
identified  in  the  next  task,  these  summaries  of  training  and  testing 
problems  become  one  of  the  primary  TEE  outputs  mentioned  auove. 

Task  E3:  Identify  task  ana  team  function  performance  discrepancies.  The  analyst 

must  first  define  an  appropriate  performance  standard  for  each  task. 
These  standards  are  based  on  task  factors  rated  in  TEE  Task  A3,  such 
as  task  criticality  or  uniqueness.  The  analyst  then  rates  test  adequacy 
for  each  task  based  on  the  seriousness  of  problem  areas  summarized 
earlier.  He  then  combines  test  ratings  with  percent  No  Go  data  for 
each  task  to  yield  a  task  rating,  either  acceptable,  discrepant,  or 
unknown.  The  analysis  process  is  conducted  to  this  point  for  both  final 
exam  and  within-course  test  data.  If  both  exist,  the  analyst  combines 
the  task  ratings  from  each  into  a  single  set  of  ratings. 

Having  determined  "combined  task  ratings"  (or  final  exam  task  ratings 
alone  when  no  within-course  tests  exist)  from  test  adequacy  and 
performance  data,  the  analyst  examines  the  presentation  data 
summaries  and  rates  the  presentation  for  each  task  based  on  the 
seriousness  of  the  problems  summarized,  again  as  acceptable, 
discrepant,  or  unknown.  Finally,  the  analyst  combines  the  ratings  from 
tests  and  performance  data  with  the  presentation  ratings  for  a  "final 
task  rating"  for  each  task.  These  ratings  represent  the  other  primary 
TEE  output  referred  to  above. 
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If  entry  or  pretest  scores  are  available,  the  analyst  can  identify  student 
selection  problems  by  arraying  entry,  pretest,  within-course,  and  final 
exam  data  for  each  trainee.  Summary  data  is  recorded  at  the  bottom 
of  the  worksheet.  Test  adequacy  is  also  rated.  In  addition,  the 
analyst  records  discrepancies  between  specified  and  actual  trainee 
characteristics. 

When  team  functions  are  analyzed,  and  data  are  available  for  their 
subordinate  tasks,  the  analyst  can  use  a  special  worksheet  to  arrange 
the  tasks  and  team  functions  hierarchically.  He  then  rates  the 
relationship  between  each  team  function  and  its  subordinate  tasks  and 
identifies  team  functions  which  are  discrepant  due  to  team# 
communication  skills  beyond  individual  task  performances  and  those 
which  are  discrepant  due  to  problems  with  subordinate  tasks. 

Phase  F:  Document  the  TEE. 

The  purpose  of  Phase  F  is  to  summarize  the  findings  of  the  TEE,  prepare  a 
report  documenting  the  effort,  and  organize  the  raw  data  worksheets  for  future  use. 
Phase  F  has  two  tasks: 

Tark  FJ:  Prepare  report  summarizing  TEE  conduct  and  findings.  Before  writing 

the  finai  report,  the  analyst  must  first  consider  whether  all  desirable 
data  have  been  collected.  If  not,  he  must  decide  whether  it  is  feasible 

to  collect  additional  data.  When  the  data  and  its  analysis  are 

complete,  the  analyst  must  then  interpret  his  findings  and  draw  some 
conclusions.  He  considers  the  following  items: 

a.  Test  adequacy. 

b.  The  number  of  tasks  rated  discrepant  or  unknown. 

c.  The  need  for  a  second  TEE  (when  the  tests  are  very  inadequate 
or  extensive  revisions  in  the  presentation  are  indicated). 

d.  Which  lessons  need  the  most  revisions. 

e.  Relationships  between  poor  lessons  and  instructors. 

f.  An  excessive  number  of  No  Cos  on  the  entry  test, 

g.  An  excessive  number  of  Cos  on  the  pretest. 

h.  Relationships  between  trainee  characteristics  and  entry  and 
pretest  scores. 

1.  Problems  with  team  functions  not  related  to  problems  with  their 
subordinate  tasks. 

The  analyst  prepares  a  rough  draft  of  his  conclusions  and 
recommendations  and  lets  some  of  his  associates  review  it,  taking  note 
of  their  questions  and  challenges.  As  the  analyst  prepares  his  complete 
report,  he  follows  sections  of  the  TEE  Handbook  which  give  examples 
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of  techniques  for  displaying  data  and  a  recommended  outline  for  the 
report. 

Task  F2:  Preserve  TEE  documentation.  So  that  future  TEE  analysts  and  those 

who  may  become  involved  in  revising  the  course  will  be  able  to 
reference  the  TEE  documentation,  the  analyst  files  it  by  task  and  step. 
The  documentation  includes  course  materials,  product  and  process  data 
collection  worksheets,  summary  worksheets,  the  final  report,  and  other 
TEE  documentation. 
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PHASE  A$  PLAN  THE  TEE 
Overview  of  Phase  A 


TASKS: 

RELEVANT  WORKSHEET: 

Al. 

Collect  the  background 
information  and  define 

TEE  purpose. 

Al 

A2. 

Review  course  materials  and 
document  task/TLO  actions. 

Task  List 

A3. 

Select  tasks  to  be  evaluated 
in  the  TEE. 

A3 

A4. 

For  tasks/TLOs  selected  for 
evaluation,  document 
conditions  and  standards. 

Task  List, 

Objectives  List 

A3. 

Specify  training  event  types 
.  and  select  events  for 
observation. 

Course  Outline 

Guidelines  for  Phase  A 

• 

Task  Alt 

Collect  background  information  and  define  TEE  purpose. 

Al.i 


Request  a  meeting  with  the  organizations  that  are  the  most 
knowledgeable  of  the  training  content  and  methods  of  the  course  that 
your  TEE  will  analyze. 


A1.2  Request  the  following  materials  and  documentation  relevant  to  the 

project: 

•  Task  Documentation  and/or  Job  Data  Worksheets 

•  ISD  or  other  documentation  relevant  to  the  training  events  that 
will  occur ,  such  as  objectives,  hierarchies,  lesson  plans,  and 
practical  exercises 

•  Test  administration  directions  and/or  any  documentation 
concerning  the  methods  that  will  be  used  during  and  at  the  end 
of  the  course  to  evaluate  the  performance  of  the  trainees 

•  Commander's  Manual  relevant  to  the  training 

•  Soldier's  ManuaKs)  relevant  to  the  training 

•  Course  materials  to  be  used  by  the  trainees 
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•  Testing/Evaluation  instruments  to  be  used  in  the  course 

•  Training/Testing  schedules 

•  Course  administration  policies 

A  1.3  Study  the  documentaton  provided  prior  to  the  initial  meeting  with 

representatives  of  the  training  organization.  Fill  out  worksheet  A1  as 
far  as  you  can. 

NOTE:  It  is  recommended  that  you  read  this  User’s  Guide  before  the  meeting 

so  that  you  will  be  familiar  with  the  requirements  of  the  TEE. 

A  1.4  Determine  the  scope  of  the  TEE  effort  in  light  of  the  TEE  purpose  and 

corresponding  resource  considerations.  As  indicated  in  Figure  2  (pg.  8), 
the  flowchart  of  TEE  tasks,  major  decision  points  determining  the  scope 
of  the  TEE  involve  the  selection  of  product  and  process  evaluation 
components  and  the  treatment  of  resulting  data.  Table  l  specifies 
those  TEE  tasks  which  are  required  and  those  which  are  optional  for  a 
TEE  in  response  to  each  of  three  basic  purposes  (i.e.,  ”OT,”  ’’Review” 
of  ongoing  training,  and  "Complaints”  arising  from  performance 
discrepancies  and/or  perceived  iastructional  deficiencies).  Optional  TEE 
tasks  are  selected  or  omitted  on  the  basis  of  time  and  manpower 
resources  available  to  the  TEE  Analyst.  For  examie  lask  B3,  Evaluate 
Presentation  Materials,  is  optional  for  a  TEE  which  has  the,  purpose  of 
reviewing  ongoing  training.  If  resources  are  constrained,  the  scope  of 
the  TEE  may  be  limited  to  a  product  and  process  evaluation  of  tests, 
rather  than  instructional  materials  as  well.  A  TEE  which  has  the 
purpose  of  responding  to  complaints  has  numerous  optional  TEE  tasks. 
This  is  because  indications  of  the  problem  area(s)  are  given  as  a 
starting  point.  If  resources  are  contained,  the  TEE  Analyst  can  thus 
focus  on  probable  discrepancies  and/or  deficiencies,  and  select  such 
components  as  product  or  process  evaluation  and  tests  or  presentation 
materials.  Note,  however,  that  a  TEE  with  the  purpose  of  fulfiJlng  OT 
requirements  has  no  optional  TEE  tasks.  This  is  because  the  TEE 
model  was  designed  specifically  for  that  purpose.  OT  training  is  newly 
developed  training,  thus  requiring  an  in-depth  analysis  of  the  entire 
training  system.  It  would  appear  then  that  a  TEE  in  response  to  OT 
requirements  can  not  be  conducted  if  resources  are  constrained. 
However,  the  scope  of  a  TEE  for  any  purpose  can  be  limited  to  a 
significant  extent  when  resources  arc  constrained  by  exercising  the 
sampling  strategies  and  contingency  options  described  within  many  of 
the  TEE  tasks.  Task  A3,  Select  Tasks  to  be  Evaluated  in  the  TEE, 
offers  the  greatest  opportunity  for  limiting  the  scope  of  a  TEE  in  this 
manner.  Additional  opportunities  exist  in  such  TEE  tasks  as  Bl,  where 
product  evaluation  questions  are  selected,  and  A5,  where  training  events 
are  selected  for  observation. 

A  1.5  Make  your  decisions  relative  to  the  scope  of  the  TEE  based  on  TEE 

purpose,  resource  considerations  and  the  goals  and  objectives  indicated 
by  your  sponsor.  At  the  initial  meeting,  you  should  identify  the 
decision  points  and  the  options  available,  and  work  with  your  sponsor  in 
defining  exactly  what  is  to  be  done.  It  is  extremely  important  that 
you  both  read)  an  understanding  of  requirements,  expectations,  and 
commitments. 
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TABLE  1.  TEE  PURPOSES  AND  CORRESPONDING  TASK  OPTIONS 


A1.6  Use  Worksheet  A1  and  notes  from  study  of  the  documentation  as  an 

agenda  for  the  initial  meeting.  Guidelines  for  completing  Worksheet  A1 
are  given  in  the  table  below. 

Guidelines  for  Worksheet  A1 


WORKSHEET  ITEM  GUIDELINES/NOTES 

Date  Date  meeting  conducted. 

Evaluator  Your  name. 

Course  Official  name  of  the  course  on  which  your  TEE 

will  be  based. 

Purpose  What  is  the  overall  goal  of  the  course? 

Primary  Audience  For  whom  was  the  course  primarily  designed? 

(Write  job  title  and/or  MOS.) 

In  addition  to  the  primary  target  audience,  are 
there  any  others  to  which  the  course  may  be 
directed?  If  so,  write  job  title(s)  or  MOS.  If  not, 
write  "none." 

Is  the  primary  target  audience  to  be  able  to  perform 
at  some  entry-level  beyond  "know  nothing  about 
the  job?”  If  so,  describe  the  expected  entry  skills/ 
knowledge  or  any  prior  training  relevant  to  the 
course  that  will  be  taken. 

Purpose  of  TEE  The  TEE  purpose  from  Step  A  1.4s  evaluate,  revise, 

or  both. 

User  of  TEE  To  whom  will  your  TEE  report  be  given?  Who  will 

act  on  your  findings-'  Who  is  the  "client”  you  are 
performing  the  TEE  for? 

User  Contact  Name  of  the  point  of  contact  you  will  work  with 

in  performing  the  TEE. 

Training  Dates/  When  and  where  will  the  training  to  be  analyzed  via 

Locations  the  TEE  be  conducted?  List  additional  dates  and 

locations  if  applicable.  Or  attach  a  schedule  of 
the  training  if  there  are  many  dates  or  locations 
for  the  training.  For  very  large  courses, 
information  should  be  entered  by  module  or  ”b!ocksH 
of  training. 


Secondary  Audience 


Expected  entry 
skills/training 


WORKSHEET  ITEM 

Testing  Dates/ 
Locations 


Training  Contact 


Limitations 


-  Resource  material 

exists? 

-  Source 

-  Date  obtained? 

-  Date  revised? 


-  Target  date  for 

completion 

-  Contingencies 


Previous  TEE 


Comparative  study? 


Training  development 
organization 


Training  delivery 
organization 


GUIDELINES/NOTES 

Often  there  is  a  considerable  time  period  between 
the  completion  of  training  and  testing  on  the  skills 
taught  in  training.  If  so,  this  time  interval  may 
be  important  in  the  diagnosis  of  training 
deficiencies  later  in  the  TEE. 

Who  will  arrange  for  your  observation  of  the  actual 
training  and  on-site  testing. 

Are  there  any  physical  or  other  conditions  in  the 
training/testing  envionment  that  will  prevent/make 
difficult  your  first-hand  observation?  If  so,  note 
them. 

-  Listed  are  the  minimum  materials  and 

documentation  needed  for  your  familiarization 
and  analysis. 

-  Add  any  others  that  are  indicated. 

-  Check  if  they  exist  or  not. 

-  Record  the  organization  who  did/will  provide 

them. 

-  Check  off  the  MObtainedM  column  when  you 

receive  each  item. 

-  Enter  the  last  date  the  materials  were  updated. 

-  When  is  your  TEE  report  due  to  be  submitted  to 

the  "client”? 

-  What  events  or  inputs  will  affect  your  meeting 

this  deadline? 

Has  a  TEE  been  conducted  on  an  earlier  version  of 
this  course?  If  "yes”,  is  that  report  available?  If 
so,  request  that  documentation  as  part .  of  the 
background  to  the  project. 

For  example,  comparing  the  performance  of  Vulcan 
crews  to  performance  of  SGT  YORK  gun  crews. 
If  so,  note  the  performance  with  which  the  present 
trainees  are  to  be  compared. 

If  the  user  of  your  TEE  is  not  the  same  as  the 
rccjuester/Hclient”  for  the  TEE,  so  note.  Request 
permission  to  contact  the  training  development 
organization  as  it  becomes  necessary  during  the 
TEE. 

Who  will  actually  conduct  the  training  on-site? 
Request  permission  to  contact  them,  as  it  later 
becomes  necessary  as  part  of  the  planning  and 
execution  of  the  TEE. 
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WORKSHEET  ITEM 
Trainees  selected? 

Entry  test? 

Instructors 

selected? 

Qualifications? 

Instructor/Trainee 

ratio 


Instructor- 

dependence 


GUIDELINES/NOTES 

On  what  basis  will  the  trainees  be  assigned  to  take 
the  course? 

Will  the  prerequisites  be  assumed  or  will  they  be 
tested  for?  If  there  is  to  be  an  entry  test,  request 
a  copy  of  it  or  any  documentation  that  describes 
the  entry  test  event. 

On  what  basis  will  the  instructors  be  assigned? 


Are  the  instructors  already  knowledgeable  in  the 
tasks  they  will  train?  Have  they  undergone 
instructor  training? 


What  will  be  the  number  of  trainees  per  instructor? 
If  there  is  a  different  ratio  in  the  classroom 
compared  to  the  field  training,  so  note.  Remember 
that  some  training  courses  exhibit  cyclical 
variations  in  the  number  of  students  per  class.  This 
may  cause  variations  in  the  trainee  to  instructor 
ratio,  if  the  number  of  trainees  remains  constant. 

Examine  the  training  plan  and  make  an  assessment 
concerning  the  degree  that  the  teaching  methods 
of  the  course  depend  on  the  skill  and  knowledge  of 
the  instructor,  as  opposed  to  relying  on  self- 
instructional  materials  and  job  performance  aids. 
Use  this  chart  as  a  guide  for  your  judgment: 


If  materials  of 
course  contain: 


A  considerable  number 
of  self-instructional 
materials  and/or  job 
aids 


Then  regard 

instructor 

dependence: 

LOW 


A  few  self -instructional  MEDIUM 

materials  and/or  job  aids 

No  self-instructional  HIGH 

materials  or  job  aids 
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WORKSHEET  ITEM 


GUIDELINES/NOTES 


After  studying  the  performance  tasks  in  your 
documentation,  make  a  judgment  about  the 
complexity  of  the  tasks  to  be  learned  in  the  course 
as  a  whole.  Use  the  chart  below  as  a  guide  for 
your  judgment: 

If  many  of  the  tasks  Then  regard 

involve:  the  tasks  as: 

A  considerable  amount  of 
visual  and  conceptual 
discrimination  and/or 
A  considerable  amount  of 
fine-grained  motor 
movements 

Discrimination  and/or  MEDIUM 

fine-grained  motor  complexity 

movements 

Mostly  straightforward  LOW 

sequential  performance  complexity 

After  studying  the  training  plan  and  discussing 
the  matter  with  the  "client",  assess  the  degree  that 
you'll  be  able  to  directly  observe  the  training  and 
testing.  Use  chart  b3ow  as  a  guide: 

If  training  and/or  The  ease  of 

testing  occurs:  observation  Is: 

Mostly  in  the  classroom  EASY 

situation 

Some  in  classroom  and  MEDIUM 

some  in  field 

Mostly  in  the  field  and/or  DIFFICULT 

inside  vehicles  such  as 

tanks 

If  the  training  Is  not  "easy"  to  observe,  get  details 
on  the  different  observation  situations  that  will  be 
encountered  in  terms  of  positions  and  tasks  that 
are  hard  to  observe.  Use  the  space  for  notes  at 
the  bottom  of  the  worksheet. 

Consider  making  preliminary  arrangements  to 
observe  some  of  those  situations.  For  example,  if 
you  must  observe  training  inside  of  an  enclosed 
vehicle  with  room  only  for  crewmembers,  you  might 
consider  arranging  to  hear  the  training  through  an 


Ease  of  training 
observation 

Ease  of  testing 
observation 


HIGH 

complexity 


Complexity  of  the 
task 


WORKSHEET  ITEM 


GUIDELINES/NOTES 


Ease  of  training 
observation 

Ease  of  testing 

observation 

(Continued) 


Mission  criticality 


Other  notes: 


open  intercom,  or  even  audio  surveillance 
equipment.  You  may  be  able  to  substitute  data 
from  trainee  reaction  questionnaires  (see  Task  Cl) 
administered  immediately  after  training  events, 
since  these  are  designed  to  ascertain  what  happened 
from  the  trainee’s  viewpoint  rather  than  to  elicit 
trainee  judgments  on  the  quality  of  the  training. 

Ease  of  observation  will  be  an  important 
consideration  in  deciding  which  factors  to  include 
on  trainee  or  instructor  reaction  instruments  in 
Steps  CL5  and  G1.7. 

Review  task  documentation  and  discuss  criticality 
of  mission  with  "client.1*  Make  as  assessment  using 
these  guidelines: 

If  the  performance 
taught  in  the  course  is:  * 

Absolutely  vital  to  the 
overall  mission  of  the 
Army  (Top  priority 
performance) 

Vital  to  the  mission  of 
unit  of  the  performer 

Necessary  to  the  mission 
of  Army  and/or  unit,  but 
below-criterion  perfor¬ 
mance  will  not  greatly 
affect  success 

Incidental  to  the  mission 
of  the  unit 

Space  has  been  left  on  the  worksheet  for  you  to 
add  additional  notes  that  may  affect  the  planning 
of  the  TEE:  coastril.iU,  important  dates,  special 
reviews,  etc. 


Then  judge 
criticality: 

VERY  HIGH 

HIGH 

MEDIUM 

LOW 
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Task  A2: 


Review  course  materials  and  document  task/TLO  actions. 


A2.1 

A2.2 

A2.3 


Familiarize  yourself  with  the  Glossary  near  the  beginning  of  the  User’s 
Guide  and  practice  classifying  CONTENT  TYPES  and  TASK  LEVELS 
using  the  materials  in  Appendix  F  if  you  have  not  already  done  so. 

You  will  recall  from  your  study  of  the  Glossary  that  a  task  is  a  ’’highly 
specific  component  of  a  job  stated  in  terms  of  a  verb  and  an  object 
with  conditions  and  standards.”  Terminal  learning  objectives  (TLOs)  are 
objectives  which  correspond  to  tasks.  Whereas  tasks  describe  job 
performance,  TLOs  are  specific  descriptions  of  the  actions, ! with 
conditions  ancf  standards,  to  be  performed  after  training. 

Sometimes  these  differences  are  small  but  they  are  still  important. 

For  example,  if  the  task  requires  the  soldier  to  decontaminate  a  piece 
of  equipment  following  a  chemical  or  biological  attack  but  the  ’’real” 
conditions  in  the  training  environment  do  not  permit  the  soldier  to  be 
subjected  to  such  a  hazard,  then  a  TLO  should  be  written  to  describe 
precisely  the  performance  which  is  acceptable  in  the  training 
environment. 

Review  all  documentation  that  is  available  and  the  information 
accumulated  via  Worksheet  Al.  Existing  course  documentation  may 
include  a  list  of  tasks  or  TLOs.  Since  the  TEE  procedures  are 
concerned  with  examining  training  rather  than  job  performance,  use  the 
list  of  TLOs  if  available. 

If  practical,  discuss  the  key  technical  training  tasks/TLOs  to  be  taught 
with  the  Instructors  or  other  personnel  knowledgeable  of  the  subject 
matter  (your  associate  analyst,  if  you  have  one). 

Develop  an  accurate  list  of  tasks  or  TLOs  and/or  team  functions.  This 
will  be  necessary  in  order  to  properly  conduct  the  TEE.  If  you  have 
obtained  one  in  Task  Al  or  if  you  can  quickly  derive  a  list  by  copying 
task  statements  appearing  in  the  manuals  or  other  documentation, 
proceed  with  Step  A2.4  below.  You  may  find  it  helpful  to  first  list 
the  tasks  or  TLOs  in  one  double-spaced  list.  Remember,  at  this  point 
in  the  TEE  you  need  only  list  the  task/TLO  action  statements,  not 
their  associated  conditions  and  standards.  These  will  be  addressed  later 
in  the  TEE  (see  TEE  Task  A4). 

If  you  cannot  obtain  a  list  of  tasks  or  TLOs  or  if  you  find  that  major 
problems  exist  with  the  actions  In  the  list  you  have,  consider  the 
options  below; 

•  If  you  can  make  revisions  to  existing  but  inadequate  tasks  or 
TLOs,  or  if  you  can  list  new  action  statements  from  manuals  and 
your  own  knowledge  of  the  course  with  relatively  little  effort, 
then  that  is  the  best  course  of  action. 

•  If  you  do  not  know  enough  about  the  course  to  revise  the  actions 
or  write  new  ones  and  you  have  access  to  a  SME,  e.g.,  your 
associate  analyst,  you  may  be  able  to  work  with  him  to  produce 
or  revise  a  task  list.  Do  this  only  if  you  are  certain  it  can  be 
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accomplished  in  the  time  available  for  this  TEE  task.  The 
listing  of  new  action  statements  is  a  long  process  for  many 
Military  Operational  Specialties.  The  feasibility  of  this  option 
will  depend  on  the  number  of  tasks  in  the  course. 

•  If  the  alternatives  above  are  not  workable,  consult  with  your 
sponsor.  It  is  not  possible  to  conduct  a  TEE  without  an 
acceptable  set  of  task  or  TLO  action  statements.  If 
arrangements  cannot  be  made  to  produce  them,  the  TEE  must  be 
postponed  until  such  arrangements  can  be  made. 

Evaluate  the  existing  task/TLO  lists  Review  each  task  or  TLO  action 
statement  using  the  Guidelines  for  Task/TLO  Actions  shown  below.  Use 
the  list  which  appears  to  be  most  adequate.  If  they  are  roughly  equal, 
however,  use  the  TLO  list,  for  reasons  noted  previously.  If  there  are 
any  problems,  make  minor  corrections  or  notes  on  your  list  at  this 
point.  Note  that  conditions,  standards,  and  other  aspects  of  the  actions 
will  be  examined  in  Task  A  4. 

Compare  the  task/TLO  list  with  operator's,  soldier's,  and/or 
commander's  manuals  to  insure  its  correctness  and  completeness.  If 
manuals  do  not  exist  or  are  outdated,  you  should  discuss  the  list  with 
instructors  or  other  subject  matter  experts  to  be  sure  it  is  up-to-date 
and  valid.  Be  sure  to  include  only  tasks  tested  in  the  course. 

If  there  are  any  major  problems  with  your  list,  consider  the  options 
listed  in  the  second  paragraph  of  Step  A2.3  above. 

Use  the  documentation  and  notes  assembled  thus  far  to  evaluate  the 
actions  the  trainee  will  be  expected  to  exhibit  after  training  -  not  the 
events  of  training,  but  the  tasks  or  TLOs  the  students  must  perform. 
Evaluate  and  revise  tasks/TLOs  and/or  team  functions  in  accordance 
with  the  guidelines  following: 

Guidelines  for  Task/TLO  Actions 

In  order  to  assess  the  adequacy  of  task  actions  (or  the  action 
component  of  a  TLO),  it  is  helpful  to  understand  the  interrelationship 
of  a  task  to  a  duty  and  to  its  job.  This  interrelationship  is  portrayed 
in  Figure  3. 

Job 

The  duties  and  tasks  performed  by  a  single  worker  constitute  his  job. 

If  identical  duties  and  tasks  are  performed  by  several  individuals,  they 
all  hold  the  same  job.  The  job  is  the  basic  unit  used  in  carrying  out 
the  personnel  actions  of  selection,  training,  classification,  and 
assignment.  In  the  Army  such  groupings  or  jobs  are  defined  as  Military 
Occupational  Specialities  (MOS).  Note  that  in  many  cases,  however,  a 
single  MOS  may  be  composed  of  several  jobs. 
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MOS 

MOS/SL 

63  B  10 

JOB 

1 

(JOB) 

TIRE  REPAIRER 

(JOB) 

WHEELED 

VEHICLE 

MECHANIC 


(JOB) 

WRECKER 

OPERATOR 


DUTIES 

l 

t 

(DUTY  A) 
ADJUSTING 

AND  REPAIRING 
BRAKES 

A 

(DUTY  B) 

TUNING  ENGINES 


(DUTY  C) 
REPAIRING 
ELECTRICAL 
SYSTEMS 


TASKS 


(TASK  B.1) 

DISTRIBUTOR 

REPAIR 


(TASK  B.2) 
REPLACE  PLUGS 


(TASK  8.3) 

REPAIR 

CARBURETOR 


(TASK  B.4) 


ELEMENTS 


ELEMENT  B.1,1) 


ELEMENT  B.3.1) 
CLEAN 

INTERNAL  PARTS 


ELEMENT  B.3.2) 
REPLACE 
WORN  PARTS 


(ELEMENT  B.3.3) 
ADJUST 
MIXTURE  JETS 


(ELEMENT  3.1.2) 


*  TO  SIMPLIFY  THIS 

EXAMPLE.  CONTINUATION 
OF  THESE  ITEMS 
HAS  BEEN  OMITTED. 


Figure  3  THE  WHEEL  VEHICLE  MECHANIC  MOS 
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DutL 


A  duty  is  one  of  the  major  subdivisions  of  work  performed  by  one 
individual.  A  job  is  made  up  of  one  or  more  duties. 

The  following  are  some  of  the  characteristics  of  duties: 

1.  A  duty  is  one  of  the  job  incumbent's  main  functions.  It 
sometimes  may  be  a  particular  job  incumbent's  total  job. 

2.  A  duty  is  a  grouping  of  closely  related  tasks. 

3.  Duty  requirements  often  are  the  basis  for  initial  assignment  to  a 
job,  for  determining  the  qualifications  required  to  perform  in  the 
job,  or  for  determining  requirements  for  post-assignment  training. 

Duties  can  usually  be  defined  by  asking  a  supervisor  what  he  thinks  are 
the  5  or  6  most  critical  factors  of  a  job  or  what  he  would  demand  of 
a  person  being  considered  for  a  job. 

Duty  titles  often  are  used  for  categorizing  groups  of  tasks  under 
identifiable  headings  to  help  in  the  organizing  of  lists  of  tasks.  At 
other  times,  duty  titles  are  assigned  for  convenience  after  tasks  have 
been  identified  and  grouped.  In  either  case,  the  duty  title  serves  to 
clearly  identify  closely  related  groups  of  tasks. 

Duty  statements  often  use  action  words  ending  in  "ing"  to  describe 
duties.  The  acton  word  generally  is  followed  by  an  object. 

Some  examples  of  duties  are: 

EXAMPLES  -  For  wheeled  vehicle  mechanics:  tuning  engines, 
adjusting  and  repairing  brakes,  repairing  exhaust 
systems,  repairing  suspension  systems,  and 
repairing  electrical  systems. 


Task 

Training  Effectiveness  Evaluation  is  accomplished  at  the  task  level.  As 
you  will  recall,  duties  are  actually  clusters  of  tasks,  the  performance 
of  which  constitute  the  duties.  A  task  is  the  lowest  level  of  behavior 
in  a  job  that  describes  the  performance  o!  a  meaningful  function  in  the 
job  under  consideration.  Task  statements  must  be  constructed  carefully 
to  assure  that  the  TEE  yields  usable  evaluation  data.  The  following 
are  characteristics  of  tasks  and  task  statements: 

1.  A  task  statement  is  a  statement  of  a  highly  specific  action. 

The  statement  has  a  verb  and  object. 

EXAMPLE  -  1.  "Repair  wheeled  vehicles"  is  not  sufficiently 

specific  to  be  a  good  task  statement.  To 
one  individual,  such  a  statement  might  mean 
performing  such  actions  as  "replace  wiper 
blades"  and  "replace  burned-out  head  lamp." 

Another  person  might  think  it  means 
"overhaul  transmission  and  engine." 
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2.  Also  "inspect  and  repair  exhaust  system" 
is  not  sufficiently  specific.  However, 

One  task  might  be  "inspect  exhaust 
system,"  and  another  task  be  "repair 
exhaust  system." 

3.  Specific  task  statements,  for  vehicle 
mechanics,  would  include:  "replace 
wiper  blades,"  "replace  burned-out 
head  lamp,"  "inspect  exhaust  system," 
and  "repair  exhaust  system." 

2.  A  Task  has  a  definite  temporal  beginning  and  end. 

EXAMPLE  -  f.  Such  action  phrases  as  "have  knowledge' 
of"  or  "take  responsibility  for"  are 
not  time-ratable  and  therefore  should 
not  be  included  in  a  task  statement. 

2.  A  task  statement  might  be  "inform 
platoon  sergeant  when." 

3.  Tasks  3re  performed  in  relatively  short  periods  of  time,  i.e., 
seconds,  minutes,  or  hours,  but  rarely,  if  ever,  days,  weeks, 
months,  or  years.  Although  no  definite  time  limit  can  be  set, 
the  longer  the  period  of  time  between  the  beginning  fcnd  the 
completion  of  the  activity,  the  greater  the  probability  that  the 
activity  is  a  generality  or  goal  rather  than  a  task. 

EXAMPLE  -  K  "To  assure  a  well-trained  army"  is 
probably  a  goal,  not  a  task. 

2.  "Perform  button-up  procedures"  takes 
a  definable  amount  of  time. 

4.  Tasks  must  be  observable  in  that  by  observing  the  performance 
of  the  job  holder  or  the  results  of  his  efforts  a  definite 
determination  can  be  made  that  the  task  has  been  performed. 


EXAMPLE  -  1.  "Understand  electronic  principles"  is  not 

observable.  Neither  the  process  nor  the 
results  can  be  observed.  (However, 
certain  actions  that  require  an  understanding 
of  electronic  principles  can  be  observed.) 

2.  An  observable  task  is  "isolate  an  error 
in  the  circuit  under  inspection." 
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5. 


A  task  must  be  measurable;  that  is,  in  the  real  world,  a 
technically  proficient  individual  can  observe  the  performance  of 
the  task  or  the  product  produced  by  the  task  and  be  able  to 
conclude  that  the  task  has  or  has  not  been  properly  performed, 

EXAMPLE  -  1.  "Know  how  to"  or  "be  able  to"  are  not 

measurable.  Neither  are  they  observable. 

2.  "Assure  success  of  mission"  is  too 
general  to  be  measurable. 

3.  "Clean  weapon"  is  measurable  and  observable.  * 

6.  Each  task  is  independent  of  other  actions.  Each  task  statement 
must  describe  a  finite  and  independent  part  of  the  job.  Tasks 
are  not^  components  of  a  procedure.  In  the  eyes  of  a  job  holder, 
a  task  is  performed  for  its  own  sake  in  the  job  situation.  A 
task  is  either  performed  or  not  performed  by  any  one  job  holder. 
The  job  holder  is  never  responsible  for  only  part  of  a  task.  If 
he  is  responsible  for  only  a  part  of  a  work  activity  that  would 
otherwise  be  defined  as  a  task,  the  part  for  which  he  is 
responsible  is  the  task. 

EXAMPLE  -  i.  If  one  of  the  wheeled  vehicle  mechanic’s 
tasks  is  "repair  exhaust  system,11  "remove 
muffler"  might  be  one  element  of  the 
task.  However,  a  helper  or  trainee  .  * 
might  be  assigned  the  task  of  "remove 
muffler."  For  the  mechanic  "remove 
muffler"  is  only  part  of  his  task.  His 
responsibility  is  not  fulfilled  until  he 
performs  the  other  appropriate  work 
dements  that  together  constitute 
"repair  exhaust  system."  However,  the 
trainee's  responsibility  for  this  particular 
work  activity  ends  as  soon  as  he  properly 
"removes  muffler."  The  wheeled  vehicle 
mechanic  does  not  remove  the  wheel  and 
tire  from  a  vehicle  for  the  purpose  of 
removing  the  wheel  and  tire.  The 
removal  is  part  of  a  procedure  intended 
to  accomplish  one  of  several  tasks  such 
as  rotate  the  tires,  repair  a  flat,  or 
install  a  new  tire.  The  latter  are  tasks 
because  they  are  independent  of  other 
actions  and  are  done  for  their  own  sake. 


Some  requirements  for  writing  good  task  statements  are  listed  in 
Table  2. 


TABLE  2.  TASK  STATEMENT  REQUIREMENTS 


Task  Statement 

Use  wording  that  is  easily 
understood. 


Be  precise  so  it  means  the 
same  thing  to  all  personnel. 


Write  separate,  specific  statements 
for  each. 

Avoid  combining  vague  items  of  skill, 
knowledge,  or  responsibility. 


Completeness  Use  abbreviations  only  after 
spelling  out  the  term. 


Include  both  form  and  title  number 
wl^er.  *e  task  is  to  complete  a  form, 
unless  all  that  is  needed  is  the 
general  type  of  form. 

Conciseness  Be  brief. 


Requirement 

Clarity 


Example 

"Compare  written  descripton 
to  actual  performance." 

But  Not 

"Relate  results  to  needs  of  field." 

Use  words  such  as  "check, 
coordinate,  assist"  with 
caution— they  are  vague. 

"Supervise  files." 

"Maintain  files." 

But  Not 

"Have  responsibility  for 
maintaining  files." 

"Inventory  War-Readiness 
Material  (WRM)"  may  be 
followed  by  "Prepare  , 
requisitions  for  WRM." 

"Complete  Task  Description 
Worksheet  (form  No.  XXX)." 


"Write  production  and  control 
reports." 

But  Not 


"Accomplish  necessary  reports 
involved  In  the  process  of 
maintaining  production  and 
control  procedures." 

Begin  with  a  present-tense  action  word  "Clean"  or  "Write." 

(subject  "I"  or  "you"  is  understood). 


Indicate  an  object  of  the  action  to  be  "Clean  engine."  "Write 
performed.  report." 


Use  terminology  that  is  currently  Use  most  recent  military 

used  on  the  job.  documentation. 


Relevance  Do  not  state  a  person's  qualifications.  "Load  computer  tape." 

But  Not 

"Has  one  year  computer  training." 

Do  not  Include  items  on  receiving  "Prepare  lab  report." 

Instruction,  unless  actual  work  is 

performed.  But  Not 


"Attend  lecture." 


34 


Element 


An  element  is  the  smallest  "package”  of  behavior  that  has  practical 
meaning  to  the  trainee.  By  "has  practical  meaning,"  we  mean  that 
further  subdivision  of  the  element  would  be  unnecessary  since  the 
trainees  fully  understand  the  element  without  further  subdivision.  To 
be  useful  as  a  basis  for  instruction,  step-by-step  direction  and  guidance 
is  required  as  to  how  the  task  is  performed.  The  work  activities  that 
make  up  this  step-by-step  direction  and  guidance  are  the  elements  that 
make  up  the  task. 

EXAMPLE  -  If  the  task  of  "perform  before-operation 

maintenance  on  2  1/2-ton  truck"  will  need  to  be 
performed,  either  individuals  must  be  trained  to  do 
the  task  or  Job  Performance  Aids  must  be  provided 
that  will  show  the  individuals  exactly  how  to 
perform  the  task.  In  either  case,  the  instructional 
developers  must  know  the  elements  that  make  up 
the  task.  These  elements  are: 

a.  Check  oil  and  coolant  levels. 

b.  Inspect  pulleys  and  fan  for  alignment,  and 
belts  fGr  tension.  Check  water  pump  and 
hose  damps  for  leaks. 

c.  Inspect  air  compressor  and  connections  lor 
security  of  mounting.  Check  belts  for 
tension. 

d.  Visually  inspect  exposed  electrical  wiring, 
conduits,  connectors,  and  shielding  for  cracks 
or  breaks. 

e.  Inspect  engine  compartment  for  indications  of 
fuel,  engine  oil,  or  water  leakage  or  seepage. 

f.  Drain  each  fuel  filter  daily  before  starting. 

g.  Check  level  of  water  in  batteries.  Check 
terminals,  clamps,  and  holddown  frames  for 
security  and  corrosion. 

h.  Check  for  loose  wheels  and  correct  tire 
pressure, 

i.  Check  general  condition  of  body  for 
scratches,  dents,  and  holes. 

j.  Inspect  cab  and  cab  body  mountings  and 
springs. 
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k.  Inspect  transfer  power  takeoff,  front  winch 
drum  lines,  drive  shaft  U- joints,  and  shear 
pins. 

l.  Check  service  break  for  proper  travel. 

m.  Notice  if  starter  pedal  requires  only  normal 
pressure  to  engage  starter  and  that  engine 
starts  immediately  without  unusual  noises. 

n.  Check  instruments  as  soon  as  you  start  the 
engine.  If  oil  pressure  stays  at  zero  or  is 
*e ry  low,  SHUT  DOWN.  Normal  ideal 
pressure  is  15  psi. 

o.  Operate  horn,  lights,  and  windshield  wipers. 

p.  Listen  for  any  unusual  noises  with  engine 
under  load. 

Without  at  least  this  level  of  detail  of  how  the  task 
is  done,  the  instructional  designer  could  not  prepare 
meaningful  instruction  or  meaningful  Job 
Performance  Aids. 

Note  that  it  is  not  necessary  to  go  down  to  the  task  elemertt  level 
during  this  TEE  task  (A2).  It  may  be  necessary,  however,  to 
decompose  tasks  to  this  level  during  a  later  task  (A4),  since  some  task 
elements  correspond  to  the  learning  objectives  (LOs)  (See  Guidelines  for 
Evaluating  Objectives). 

If  an  existing  task/TLO  list  contains  task  elements,  you  can  ignore 
them  for  now.  feemember,  it  is  the  task  action  which  you  are 
concerned  with  during  this  step. 

Your  task  actions,  in  addition  to  being  appropriate  "chunks"  of  behavior 
and  well-written,  should  also  conform  to  the  following  guidelines: 

•  Are  task  actions/TLOs  written  for  each  critical  task?  Critical 
tasks  have  the  following  characteristics: 

% 

They  have  serious  consequences  of  inadequate  performance 
in  terms  of  their  impact  on  task  performance. 

-  They  are  common  sources  of  failure. 

They  are  actions  which  the  soldiers  of  the  target 
population  do  not  already  know  how  to  do  and  must 
therefore  be  taught. 

•  Are  subordinate  tasks/objectives  common  to  more  than  one  higher 
task/TLO  or  team  function  so  identified? 


•  If  the  course  involves  functions  performed  by  a  TEAM,  is  there  a 
list  or  hierarchy  showing  each  TEAM  FUNCTION  with  its 
subordinate  tasks/TLOs? 

•  If  skills  such  as  coordination,  cooperation,  or  communication  are 
critical  elements  of  TEAM  FUNCTIONS,  are  they  appended  as 
tasks/TLOs  subordinate  to  the  functions  in  those  lists  or 
hierarchies? 

•  Is  an  appropriate  numbering  system  used  with  which  to  reference 
tasks,  TLOs,  and  TEAM  FUNCTIONS? 

Tasks  can  be  numbered  either  serially  (e.g.,  24,  25,  26, 
etc.)  or  hierarchically  (e.g.,  24,  24.1,  24.2,  24.3,  etc.) 

TEAM  FUNCTIONS  can  be  assigned  numbers  higher  than 
the  last  task  numbers  on  the  task  list. 

TLOs  should  be  numbered  to  correspond  with  the  task  list. 
For  example,  if  a  job  performance  task  is  numbered  "177", 
the  corresponding  TLO  could  be  numbered  177T  to  indicate 
that  it  is  the  training  (T)  equivalent  of  Task  177. 

Skills  such  as  coordination  and  communication  can  be 
designated  with  letters. 
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Figure  ft.  Example  Task/TLO/Team  Function  numbering  system 


In  the  above  example,  tasks  are  numbered  serially.  Team  functions  are 
assigned  3-digit  numbers  to  distinguish  them  from  individual  tasks  (l- 
and  2-digit  numbers).  TLOs  26T  and  27T  are  the  training  equivalents 
of  Job  Tasks  26  and  27.  Coordination  and  communication  skills  are 
designated  with  letters  (A  and  B). 

o  Further  information  on  task  listing  and  analysis  may  be  found  in 
the  following  references: 

Schultz,  R.E.  6c  Farrel,  J.R.  Job  aid  manuals  for  Phase  I- 
ANALYZE  of  the  instructional  systems  development  model 
(Research  r  od'srt  SO- lft).  Alexandria,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  1930. 

U.S.  Army  Headquarters,  United  States  Army  Training  and 
Doctrine  Command,  fntcrservice  procedures  for  iretructional 
systems  development.  Phase  I  -  Analyze  (TRAPOfc  Pam  330-30). 
Fort  Monroe,  VA:  Author,  1973. 

U.S.  Army  Headquarters,  United  States  Army  Training  and 
Doctrine  Command.  Job  and  Task  Analysis  Handbook  (TRADOC 
Pam  351-ft(T)).  Fort  Monroe,  VA:  Author,  1975. 
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Task  A3:  Select  tasks  to  be  evaluated  in  the  TEE. 

A3.i  Decide  whether  or  not  you  can  evaluate  all  of  the  tasks  in  the  course. 

The  course  you  will  evaluate  in  the  TEE  may  teach  a  small  number  of 
tasks.  That  is,  it  may  be  so  short  that  you  can  perform  the  TEE  on 
all  the  tasks  being  dealt  with  in  the  course.  More  likely,  the  course 
will  deal  with  a  large  number  of  tasks.  If  so,  it  is  usually  impractical 
to  attempt  evaluation  of  all  tasks  being  taught.  If  that  is  true,  it  is 
likely  that  a  TEE  covering  all  tasks  is  beyond  your  resources.  If  so, 
select  tasks  for  evaluation  beginning  in  Step  A3.1  below.  If  the 
necessity  for  task  selection  is  not  obvious,  then  make  a  judgment 
considering  the  following  points: 

o  Each  task  is  trained  using  a  series  of  training  events,  such  as  a 
lecture,  a  demonstration,  hands-on  practice,  and  a  test.  Training 
for  a  task  may  also  include  events  for  objectives  subordinate  to 
the  overall  task. 

•  You  should  be  able  to  form  an  idea  of  the  average  number  of 
training  events  per  task,  how  long  they  take,  and  what  you  would 
have  to  do  to  observe  and  evaluate  them.  Consider  the  fact 
that  you  will  have  to  conduct  an  evaluation  of  the  training 
events  as  they  occur  in  the  course,  as  well  as  an  evaluation  of 
the  presentation  materials  and  tests. 

•  Consider  the  time  you  have  to  conduct  the  TEE;  the  number  of 
personnel  at  your  disposal;  and  the  logistics  of  the  training*  i*e., 
number  of  classes  and  how  their  schedules  may  intertwine  for 
different  tasks. 

•  Make  a  judgment  as  to  whether  the  whole  course  can  be  easily 
evaluated  within  resources.  If  so,  skip  this  task  (A3)  and  go  on 
to  task  A4.  If  it  is  not  feasible  continue  with  Step  A3.2  below. 

Guidelines  for  Worksheet  A3 

A3.2  Find  Worksheet  A3.  Record  the  reference  numbers  from  your  task  list 

in  the  Task  ID  0  column.  Ask  each  question  in  Steps  A3.3  through 
A3.8  of  all  tasks  on  your  list,  as  directed. 

Data  for  the  task  factors  below  may  be  contained  in  the  task 
documentation.  You  have  the  option  of  accepting  those  ratios  or 
using  your  own  judgment  in  conjunction  with  the  guidance  for  this  step, 
if  there  arc  no  task  data,  and  you  feel  you  are  not  qualified  to  make 
these  estimates,  consult  with  stiject  matter  experts. 

IMPORTANT;  After  assessing  your  entire  task  list  on  the  factor  in 
each  of  Steps  A3.3,  A3.4,  A3.6,  A3.7,  arid  A3.8,  go  to  Step  A3.9  and 
eliminate  tasks  on  that  factor.  (Tasks  are  not  eliminated  on  the  factor 
in  Step  A 3.5  by  itself.)  Then,  judge  whether  you  can  conduct  the  TEE 
on  the  number  of  tasks  remaining  at  that  point.  If  so,  you  need  not 
evaluate  the  tasks  in  the  steps  you  have  remaining  in  this  task  (A3). 
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In  that  case  go  on  to  Task  A4.  Only  if  too  many  tasks  remain  on  the 
list  to  conduct  the  TEE  should  you  go  on  to  the  next  step  in  this  task 
after  eliminating  tasks  each  time. 

.3  Some  tasks  are  part  of  the  overall  performance  to  be  learned  in  the 

course,  but  the  majority  of  the  trainees  already  have  learned  the  task 
previously.  That  is,  they  enter  the  course  with  an  acceptable  level  of 
proficiency.  For  example,  students  in  training  for  an  Operational  Test 
of  a  new  weapon  system  may  have  been  selected  from  an  MOS  similar 
to  the  new  weapon  system  MOS.  As  a  result  they  may  already  be 
proficient  on  many  tasks  on  the  new  system. 


IF  THE  TASK  IS: 

THEN  RECORD: 

Already  KNOWN  by  the 
majority  of  the  trainees. 

"YES”  in  the  second 
column  of  worksheet  A3. 

NOT  known  by  the  majority 
of  the  trainees  on  entry 
to  the  course. 

"NO"  in  the  second 
column  of  worksheet  A3. 

NOTEi  The  "KNOWN”  factor  will  not  apply  if  your  TEE  is  limited 
to  the  evaluation  of  tests  and/or  performance  scores. 

4  Make  an  assessment  concerning  the  likelihood  of  performance 

PROBLEMS  for  each  task.  This  factor  is  a  combination  of  how  often 
below-standard  performance  occurs  on  the  job  for  each  actual  task  or 
similar  tasks  and  how  difficult  it  Is  to  learn  the  task. 

(Consult  subject  matter  experts,  if  necessary.) 


IF  TASK  IS: 

AND/OR: 

THEN  RECORD: 

Known  to  have  had 
performance  problems 
in  the  past. 

Difficulty  In  becoming 
proficient  in  the  past. 

"YES”  in  the  third 
column. 

NOT  known  to  have  Task  task  seems  to  be 
had  performance  difficult  to  learn.* 

problems  In  the  .  . — 

past.  The  task  seems  to  be 

relatively  easy*  to 
learn. 


^"Difficult”  or  "easy”  to  learn  is  usually  a  matter  of  the  characteristics 
of  the  performance  and/or  the  presence  of  very  high  accuracy  or  speed- 
of-performance  standards.  Some  generalizations; 


"MAYBE”  in  the  third 
column. 

”NO”  in  the  third  column. 


a  3.5 


NOTE: 


o  Tasks  requiring  fine  discrimination/decisions  are  usualiy 
"difficult.11 

o  Tasks  requiring  a  few  simple  steps  are  usualiy  "easy." 

o  Tasks  requiring  fine  tolerance,  psychomotor  movements  are 
usually  "difficult." 

o  Tasks  requiring  many  steps  are  usually  difficult  to  remember  - 
thus  difficult  to  learn,  if  not  to  perform. 

If  you  are  conductng  a  TEE  for  an  Operational  Test  (OT),  you  should 
stop  at  this  point  and  go  to  Step  A3.9  to  seiect  tasks  for  evaiuation. 
None  of  the  remaining  factors  in  Steps  A3.5  through  A3.7  are 
appropriate  for  the  OT  setting.  If  there  are  still  too  many  tasks  to 
evaluate,  confer  with  your  sponsor  to  discuss  possible  approaches.  You 
may  wish  to  discuss  the  criticality  of  each  task  in  terms  of  its 
Importance  to  the  Operational  Test  and  construct  a  scale  similar  to  the 
one  in  Step  A3.5.  Then,  for  each  task,  make  a  judgment  concerning 
how  critical  it  is. 

For  each  task,  make  a  judgment  concerning  how  critical  it  is.  That  is, 
how  serious  are  the  consequences  of  not  performing  the  task  adequately 
under  real  job  conditions?  Use  the  rating  scale  below  for  task 
criticality. 

* 

4  c  Critical.  Failure  to  perform  the  task  to  criterion  will 
result  in  casualties  and/or  complete  failure  of 
the  mission. 

3  s  Very  Important-  Essential  task,  but  failure  will  not 

necessarily  result  in  casualties  or  failure  of  the 
mission. 

2  -  Moderately  important*  Performance  1$  of  significant 
value,  but  failure  will  not  jeopardize  the 
mission. 

1  55  Not  important.  Performance  is  useful,  but  failure  is 

incidental  to  success  of  the  mission. 

Criticality  is  to  be  used  in  conjunction  with  ‘•uniqueness"  and  "use"  in 
Step  A3.9;  therefore  go  to  step  A3.o  at  this  point  without  branching  to 
Step  A  3.9. 
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A3.6 


Make  an  assessment  of  the  UNIQUENESS  of  the  training  of  each  task, 
i.e.,  how  easy  it  is  to  learn  the  task  on  the  job.  Uniqueness  is  a 
function  of  the  three  factors  considered  in  the  box  below. 


IF: 

AND: 

AND: 

THEN  RECORD: 

There  is  little 
time  to  learn  the 
task  after  an 
assignment  is 
given  to  perform 
the  task. 

The  task  must 
typically  be 
performed  soon 
after  assignment 
to  the  jpb. 

The  task  has  proven 
difficult  to  learn 
in  the  past  or 
seems  difficult 
to  learn.* 

"YES"  in  the  fourth 
column  of  A3. 

OTHERWISE: 

"NO"  in  the  fourth 
column  of  A3. 

It 

See  guidance  on  learning  difficulty  under  Step  A3.4  above. 


A3.7  Some  task  skills  are  used  on  the  job  more  than  others.  Make  an 

assessment  of  task  USE. 


IF  TASK: 

AND/OR: 

AND: 

THEN  RECORD: 

Is  performed 
frequently  on 
the  job. 

A  large  percen¬ 
tage  of  time  on 
the  job  is  spent 
on  the  task. 

Performed  by  most 
MOS  holders  in 
the  field. 

"YES"  in  the  fifth 
column. 

■  Performed  by 
many  MOS  Holders 
in  the  field. 

•5 

"MAYBE"  in  the 
fifth  column. 

OTHERWISE: 

"NO"  in  the  fifth 
column. 

A3.8  Use  the  space  in  the  OTHER  column  on  Worksheet  A3  to  enter  any 

other  factors  or  notes  relevant  to  the  decision  whether  to  include  the 
task  in  the  TEE  or  not.  For  example,  some  tasks  may  be  required  to 
be  part  of  the  TEE  by  directive  or  by  request  of  your  sponsor  or  other 
user  of  the  TEE. 


A3.9 


Use  the  table  below,  a  row  at  a  time,  after  each  of  the  above  steps 
you  perform  (except  A3.5).  Check  the  ELIMINATE  column  on 
Worksheet  A3  as  indicated.  All  tasks  not  eliminated  are  assumed  to  be 
retained. 
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If  there  are  still  too  many  tasks  to  evaluate  after  making  the 
eliminations  for  the  step  you  just  completed,  go  back  and  continue  with 
the  next  step. 


IF: 

KNOWN 

column  is  "YES". 

PROBLEMS 
column  is  "NO". 


AND: 


Eliminating  via 
KNOWN  still 
leaves  too  many 
tasks  to  analyze. 


AND: 


THEN: 

ELIMINATE 

ELIMINATE 


The  remaining  factors  below  are  not  appropriate  for  a  TEE  of 

an  Operational  Test. 


UNIQUE  column 

Eliminations  via 

CRITICALITY 

=  1 

ELIMINATE 

IS  "NO”. 

KNOWN  and 
PROBLEMS  still 

CRITICALITY 

=  2 

Consider  eliminat¬ 

leave  too  many 

ing.*  (Mark  "C"  or 

tasks  to  analyze 
in  the  TEE. 

"Consider”) 

CRITICALITY 

=  3 

RETAIN  - 

or  4 

USE  column  is 

Eliminations  via 

CRITICALITY 

s  1 

ELIMINATE 

"NO". 

all  of  the  factors 
above  still  leave 

CRITICALITY 

-  2 

Consider  eliminat¬ 

too  many  tasks 

ing.  (Mark  "C” 

to  analyze  in 

or  "Consider”) 

the  TEE. 

CRITICALITY 

=  3 

RETAIN 

or  4 

*If  there  are  stll  too  many  tasks  for  the  TEE  after  the  task  list  is 
reduced  by  the  tasks  eliminated  in  this  step,  (but  not  by  those  marked 
by  consideration),  go  on  to  step  A3 .7  and  consider  USE. 


NOTE:  If  the  considerations  above  still  leave  too  many  tasks  for  practical 

analysis,  given  your  time  and  resources,  then  you  must  either  request 
additional  time/resources  or  consult  with  your  sponsor  to  select 
additional  tasks  for  elimination.  Think  of  any  tasks  marked  to  consider 
for  elimination  as  primary  candidates. 


A3. 10 


Delete  tasks/TLOs  not  selected  for  evaluation  from  your  task/TLO  list. 


Task  A4:  For  tasks/TLOs  selected  for  evaluation,  document  conditions  and 

standards,  and  evaluate  objectives. 

The  purpose  of  this  task  is  to  document  objectives  so  that  they  validiy 
reference  tasks  as  they  are  performed  on  the  job.  Therefore,  the 
documentation  in  the  course  cannot  be  your  only  reference,  since  you 
are  evaluating  its  adequacy'.  You  will  need  to  consult  subject  matter 
experts  to  insure  that  the  tasks  or  objectives  you  use  for  the  TEE  are 
valid  and  up-to-date. 

This  is  not  likely  to  be  a  trivial  task.  Only  if  (a)  tasks  or  TLOs 
already  exist,  Cb)  the  TEE  is  to  be  conducted  at  that  gross  level,  and 
(c)  they  need  little  . or  no  revision,  will  your  task  be  relatively  simple. 

It  could  take  days  to  evaluate  the  large  mass  of  finer  learning 
objectives  (LOs)  (See  Step  A4.2)  that  may  exist  for  a  course  of  any 
size,  even  if  there  are  no  problems  with  them.  Writing  new  tasks  and 
objectives  for  a  course  can  take  months.  Therefore,  proceed  with 
caution  and  carefully  consider  the  alternatives  given  if  you  should  run 
into  any  problems. 

A4.1  Consider  options  when  there  are  no  conditions  and  standards. 

If  there  are  no  conditions  or  standards  in  your  task/TLO  list,  consider 
some  alternatives.  (Note  that  LOs  are  dealt  with  in  Step  A4.2). 

o  If  you  can  add  conditions  and  standards  from  manuals  and  your 
knowledge  of  the  course  with  relatively  little  effort,  that  is  the 
most  desirable  course  of  action. 

o  If  you  do  not  know  enough  about  the  course  to  add  conditions 
and  standards,  you  may  wish  to  consider  working  with  a  subject 
matter  expert  to  resolve  existing  problems.  Do  this  only  if  you 
are  certain  it  can  be  accomplished  in  the  time  available  for  this 
task.  (It  can  be  a  lengthy  process  as  noted  above.)  You  may 
wish  to  consult  your  sponsor  about  limiting  the  number  of  tasks 
to  evaluate  further,  if  it  can  make  this  alternative  feasible.  To 
do  so,  continue  with  the  process  begun  in  Task  A3. 

o  It  is  not  possible  to  conduct  the  TEE  without  conditions  and 

standards.  If  the  alternatives  above  are  not  workable,  however, 
you  can  proceed  with  task  or  TLO  actions  alone,  but  you  and  ail 
of  your  data  collectors  must  be  thoroughly  familiar  with  the 
conditions  and  standards.  Training  on  tasks  by  a  subject  matter 
expert  is  included  in  Task  Dls  Train  Data  Collectors. 

A4.2  Locate  any  additional  documentation  which  contains  the  objectives  of 

the  course  to  the  greatest  level  of  detail  available.  Ideally  you  should 
have  TLOs,  which  correspond  to  tasks,  and  learning  objectives  (LOs) 
which  match  the  major  subtasks.  Read  the  General  Guidance  on 
Objectives  below  as  a*.  Md  to  identifying  the  appropriate  objectives. 
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General  Guidance  on  Objectives 


•  For  each  task  statement  one  terminal  learning  objective  (TLO) 
should  exist.  TLOs  can  be  broken  down  further  into  enabling 
objectives  called  learning  objectives  (LOs).  You  will  recall  that 
a  learning  objective  (LO)  is  a  subunit  of  a  terminal  learning 
objective  consisting  of  a  precise  description  of  what  is  to  be  • 
learned  in  terms  of  the  expected  student  performance  under 
specified  conditions  to  accepted  standards.  These  learning 
objectives  identify  the  mental  skills,  information,  attitudes,  or 
physical  skills  that  are  required  to  perform  the  terminal  learning 
objective.  Each  LO  can  be  broken  down  further  to  as  many 
levels  of  detail  as  are  necessary  due  to  the  task’s  complexity. 

e  TLOs  and  LOs  can  be  arrayed  in  a  learning  or  objectives 
hierarchy.  A  learning  hierarchy,  or  shorthand  pyramid  of 
objectives  is  a  quick  and  efficient  way  of  showing  a  working 
draft  of  the  skills,  knowledges,  and  their  relationships  to  one 
another,  necessary  for  performing  the  terminal  objective. 

Through  such  a  hierarchy,  the  analyst  creates  a  visual 
representation  of  the  relationship  between  a  terminal  objective 
and  its  subordinate  objectives. 

•  Figure  5  illustrates  an  example  learning  hierarchy.  In  this 
hierarchy,  the  analyst  has  taken  one  specific  task  from  the  task 
list  (navigate  from  point  A  to  point  B)  and  has  written  a  task 
statement  (task/condition/standard)  for  that  portion  of  the  course. 
From  the  terminal  objective,  he  has  derived  the  major 
subordinate  skills  (LOs)  (orient  map  and  compass/hike  with  pack), 
and  then  further  broken  down  those  subordinate  skills  into  lower 
level  prerequisite  skills  and  knowledges.  Further,  he  has  shown, 
by  drawing  prerequisite  lines,  the  relationships  between  all 
subordinate  skills  and  knowledges  and  the  terminal  objective. 

Thus,  for  purposes  of  course  evaluation,  he  can  now  determine 
which  specific  skills  must  be  learned  before  the  student  can 
master  higher  level  skills.  With  the  completed  learning  hierarchy 
there  is  a  graphic  representation  of  the  essential  content  of  this 
particular  portion  of  the  course. 

•  In  a  complete  analysis,  objectives  would  be  written  down  to  the 
level  just  above  a  point  where  it  can  be  assumed  that  anyone 
with  the  capacity  to  be  trained  in  the  task  could  perform  the 
objectives. 

Example  of  an  objective  any  trainee  could  perform; 

For  a  task  involving  a  PATRIOT  operator’s  console: 

Identify  shapes  (a  necessary  skill  in  locating  controls). 


45 


READ  VALUES 
W/INTERPOL. 


APPLY  RULES 
FOR  DIRECTION 


IDENTIFY 

SYMBOLS 


IDENTIFY* 

GRADUATION 

MARKS 


I.D.  ZERO 
REFERENCE 
|  POINT 


I.O.  BLUE 
POINTER 


IDENTIFY 

SHAPES 


IDENTIFY 

COLORS 


O 


-  TO  SIMPLIFY  THIS  EXAMPLE. 
CONTINUATION  OF  THESE 
ITEMS  IS  OMITTED. 


Figure  5  LEARNING  HIERARCHY 
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•  Learning  Hierarchy  Numbering  System 

The  major  task  objectives  which  form  the  top  boxes  in  the 
hierarchies  for  a.  course  are  assigned  the  exact  same  reference 
numbers  as  they  are  given  in  the  task  listing.  Each  subordinate 
objective  in  a  hierarchy  has  a  reference  number  which  includes 
the  complete  reference  number  from  its  superordinate  objective 
in  the  hierarchy  and  an  added  entry.  For  example,  a  single  LO 
subordinate  to  Task  26  would  be  numbered  26.1. 

If  a  number  of  objectives  are  immediately  subordinate  to  the 
same  objective,  the  reference  numbers  for  all  of  them  will  be 
identical  in  all  but  the  last  entry.  The  final  entries  for  the 
subordinate  objectives  will  be  sequentially  assigned  numbers 
beginning  with  1.  For  example,  in  the  example  hierarchy,  Figure 
6  below,  the  subordinate  objectives  to  objective  3.2  are  numbered 
3.2.1,  3.2.2,  3.2.3,  3.2.4,  and  3.2.5. 


Figure  6  EXAMPLE  LEARNING  HIERARCHY  NUMBERING  SYSTEM 


•  At  some  point  above  the  lowest  level  of  objectives  which  ought 
to  be  written  in  a  complete  analysis,  a  line  can  be  drawn  (not 
necessarily  a  straight  line),  below  which  objectives  are  classed  as 
entry  skills.  An  entry  skill  is  something  a  student  will  be 
expected  to  already  know  or  be  able  to  perform  prior  to  entering 
the  course.  The  objectives  above  the  line  are  to  be  taught  in 
the  course. 

•  Terminal  learning  objectives  should  be  identical  to  task 
statements  except  when  the  task  conditons  or  standards  are  so 
dangerous,  costly,  or  time-consuming  that  the  conditions  cannot 
be  simulated  or  the  standards  cannot  be  required  in  the  intended 
training  environment.  How  well  the  objectives  match  the  tasks 
is  referred  to  as  fidelity.  Each  TLO  should  have  the  greatest 
fidelity  possible,  except  when  it  can  be  shown  that  tests  based 
on  lower  fidelity  objectives  can  accurately  predict  the  real  task 
performance. 

For  example,  an  objective  that  is  simple  in  terms  of  physical 
movements,  but  that  requires  a  great  deal  of  mental  expertise, 
might  have  conditions  specifying  a  paper  and  pencil  test  if  it 
could  be  shown  or  realistically  assumed  that  those  who  could 
pass  the  test  could  also  perform  the  task  and  that  those  not 
passing  the  test  could  not  perform  the  task. 

If  the  smallest  unit  of  instruction  addressed  by  any  test  item  or 

training  event  is  an  LO,  or  you  have  a  complete  set  of  adequate  LOs, 

skip  this  step  and  go  on  to  step  A4.4.  If  the  course  addresses  LOs  and 

a  complete  set  of  LOs  does  not  exist,  consider  the  following  options: 

•  If  you  can  make  revisions  or  list  new  LOs  from  manuals  and 
your  own  knowledge  of  the  course  with  relatively  little  effort, 
that  is  the  best  course  of  action. 

•  If  you  do  not  know  enough  about  the  course  to  revise  LOs  or 
write  new  ones  and  you  can  have  access  to  a  subject  matter 
expert,  you  may  be  able  to  work  with  him  to  do  so.  *  Do  this 
only  if  you  are  certain  it  can  be  accomplished  in  the  time 
available  for  this  task  (A4).  The  listing  of  new  LOs  is  an 
especially  long  process  for  many  MOSs.  The  feasibility  of  this 
option  will  depend  on  the  number  of  LOs  in  the  course. 

•  If  neither  of  the  above  options  is  feasible,  but  could  be  made  so 
if  the  number  of  tasks  in  the  evaluation  were  limited  further, 
consider  continuing  the  selection  process  in  Task  A3  until  one  of 
these  approaches  can  be  used.  Consult  with  your  sponsor  to  see 
if  he  prefers  this  approach  or  one  of  the  next  two. 

•  The  TEE  can  be  conducted  at  the  TLO  level  by  asking  questions 
regarding  the  match  of  test  items,  objectives,  and  presentation 
materials  and  events  at  the  TLO  level  rather  than  the  LO  level. 

•  If  you  have  a  list  of  TLOs  and  LO  actions,  you  can  conduct  the 
TEE  without  LO  conditions  and  standards  if  you  are  thoroughly 
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familiar  with  the  difference  between  the  conditions  and 
standards  in  the  job  setting  (referenced  by  tasks  and 
elements)  and  the  learning  setting  (referenced  by  TLOs  and 
LOs),  and  have  an  idea  of  what  the  task  elements  are. 

With  that  knowledge  you  (or  your  associate  analyst)  should 
be  able  to  make  the  judgments  required  in  the  TEE 
questions  that  deal  with  LOs.  If  this  is  your  decision,  and 
you  are  not  a  subject  matter  expert,  you  will  need  to 
study  the  conditions  and  standards  in  the  tasks  and  TLOs. 
Sit  down  with  a  subject  matter  expert  and  discuss  tve 
conditions  and  standards  in  both  settings  for  the  LOs  and 
make  some  detailed  notes  before  proceeding.  Any  task 
documentation  containing  task  elements  should  be  studied, 
since  the  elements  often  have  the  same  actions  as  the 
most  specific  LOs. 

AM  Evaluate  the  tasks/TLOs  (if  you  have  not  already  done  so)  and  existing 

LOs  using  the  Guidelines  for  Evaluating  Objectives  below.  Make 
editorial  notes  or  comments  on  your  list  of  objectives  as  you  go  along. 
Be  aware  that  when  dealing  with  LOs,  you  can  quickly  exceed  your 
resources  if  the  LOs  are  not  in  good  shape.  When  you  have  completed 
this  step  (AM),  under  ideal  conditions  you  will  have  an  acceptable  set 
of  TLOs  and  LOs  that  match  the  job  tasks.  They  are  the  standard 
with  which  you  will  judge  the  course  of  instruction. 

Guidelines  for  Evaluating  Objectives  (GEO)  : 

For  each  TLO  and  LOs  at  every  level  of  detail  ask  each  of  the 
questions  in  Sections  1-3  below.  Edit  or  note  problems  with  objectives 
as  required.  If  no  entry  test  is  needed  for  the  course,  you  need  not  be 
concerned  with  objectives  below  the  entry  level.  The  questions  In  each 
section  should  be  used  with  the  set  of  objectives  supporting  each  TLO 
before  going  on  to  the  next  TLO.  TEAM  FUNCTIONS,  as  defined  in 
Section  4,  should  also  follow  the  rules  in  Sections  1-3. 

Use  existing  manuals,  other  documentation,  and  subject  matter  experts 
as  your  resources.  Do  not  use  course  t^sts,  however,  as  their  validity 
is  to  be  assessed  by  comparing  them  wi. .»  the  objectives  which  are 
matched  to  the  tasks  in  this  step. 

In  evaluating  the  LOs,  any  documentation  of  task  elements  will  be 
valuable,  since  the  LO  action  statements  will  often  be  the  same  for 
LOs  at  the  greatest  level  of  detail. 

In  the  next  few  pages,  the  criteria  for  determining  the  adequacy  of 
LOs  are  listed.  Following  this  section  is  an  EXAMPLE  section,  in 
which  several  sample  objectives  are  rated  and  explanations  for  the 
ratings  are  provided. 
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Objective  Adequacy 


1. 


Each  objective  must  be  correctly  stated  in  terms  of  CONDITIONS, 
STANDARDS,  and  the  ACTION  the  student  must  perform. 


Several  categories  of  conditions  and  standards  are  listed  below.  Obviously, 
no  objective  will  require  all  of  these.  Each  objective  should  be  reviewed 
with  these  categories  in  mind,  and  a  decision  should  be  made  about  whether 
or  not  they  are  applicable.  If  you  are  unsure  about  whether  or  not  a 
particular  condition  or  standard  should  be  included  in  an  objective,  a  good 
rule  is  "when  in  doibt,  stick  it  in." 

> 

CONDITIONS 


Environment  Information 


Resources 


Physical 

Social 

Psychological 


Given 

Cues  or  Dues 
Special  Instructions 


3ob  Aids 

Equipment  or  Tools 
Technical  Manuals 


STANDARDS 


Performance 


Product 


Completeness 

Accuracy 

Time  Limit  or  Rate 


Completeness 

Quality 

Judgment 


NOTE;  Many  objectives  contain  "implicit1'  conditions  or  standards  like  "Given  paper 
and  pencil,  ..."  or  "with  100%  accuracy."  Obviously  the  action  part  of 
an  objective  should  use  an  action  verb.  It  is  usually  best  that  there  be 
only  one  action  per  objective;  if  there  is  more  than  one  action,  the 
objective  should  probably  be  split  into  several  objectives.  The.  action  verb 
should  always  be  observable  and  measurable. 

2.  Each  objective  must  be  classifiable  by  TASK  LEVEL  and  CONTENT  TYPE. 
A  classification  matrix  is  given  in  Figure  7.  If  an  objective  cannot  be 
classified,  it  must  be  restated.  If  an  objective  fits  more  than  one  cell  of 
the  matrix,  it  probably  needs  to  be  split  up  into  more  than  one  objective. 


3.  Each  objective  must  be  appropriate  in  the  following  ways; 

a.  Conditions,  standards,  and  the  task  level  and  content  type  of  the 
action  must  be  appropriate  for  the  work  to  be  performed  on  the 
job  or  for  later  training. 


b.  If  the  objective  is  classified  as  a  REMEMBER  task  level,  there 
should  be  a  later  USE  objective. 

c.  If  the  objective  is  classified  as  a  USE-UNAIDED  task  level,  there 
should  be  a  previous  REMEMBER  objective. 


d. 


If  the  objective  is  classified  as  a  USE- AIDED  task  level,  the  aid 
must  be  adequate,  or  other  objectives  on  the  aid  must  be  included. 
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Test  Consistency 

1.  The  CONDITIONS  in  each  test  item,  or  the  conditions  under  which  the 
items  are  administered,  must  match  the  conditions  in  the  objective. 

2.  The  STANDARDS  in  each  test  item,  or  the  standards  for  scoring  each 
item,  must  match  the  standards  in  the  objective. 

3.  The  ACTION  in  each  test  item  mus+  ^atch  the  action  of  the  objective 
in  terms  of  task  level  and  content  type. 

4.  The  FORMAT  for  each  test  item  must  be  appropriate  for  the  task  level 
and  content  type  as  indicated  in  the  following  section. 

Test  Adequacy 

1.  Each  test  item  must  be  clear.  Instructions  for  completing  the  item  must 
specify  what  response  the  student  is  expected  to  make. 

2.  Each  test  item  must  be  unambiguous.  Each  item  must  have  one  and  only 
one  correct  response,  and  the  item  must  be  interpretable  in  only  one  way. 
Items  must  not  be  confusing. 

3.  Each  test  item  must  be  free  of  hints.  An  item  should  not  give  away  the 
answer  to  itself  or  to  any  other  item  on  the  test.  The  grammar  of 
multiple-choice  and  fill-in  items  should  not  give  hints  to  answers. 

4.  Test  items  should  allow  common  errors  to  be  made. 

5.  Enough  test  items  must  be  provided  for  USE-LEVEL  objectives  to  test  the 
objective  adequately,  and  to  reflect  the  range  of  performance  required  on 
the  job. 

6.  Each  test  item  must  be  well  constructed.  Different  criteria  apply  to 
different  item  formats  as  indicated  below! 

TRUE-FALSE 

An  item  should  include  only  one  statement  to  be  judged  true  or 

false. 

Negative  statements  should  be  avoided. 

Do  not  use  words  like  "never "always,"  etc. 

Item  statements  should  be  short. 
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MULTIPLE  CHOICE 


All  alternatives  should  be  plausible. 

Negatives  In  the  item  stem  should  be  highlighted. 

Repetitive  phrases  should  be  placed  in  the  stem,  not  in  the 
alternatives. 

Alternatives  like  "All  of  the  above’1,  "A  and  B  only,'1  should  be 
avoided. 

MATCHING 

Instructions  should  explain  the  contents  of  each  column,  and  explain 
the  basis  for  matching. 

Instructions  should  specify  how  many  times  each  answer  may  be 
used. 

The  choice  column  should  include  extra  answers  unless  answers  may 
be  used  more  than  once. 

FILL-IN 

The  blank  should  be  at  or  near  the  end  of  the  sentence. 

One  and  only  one  word  or  phrase  should  correctly  complete  the 
item. 

Blanks  should  require  key  words. 

SHORT  ANSWER 

The  required  answer  should  be  kept  short. 

The  directions  to  the  student  should  specify  how  the  item  will  be 
scored. 

The  scoring  key  should  identify  allowable  synonyms  or  alternatives. 
LISTING 

The  directions  should  specify  the  number  of  items  to  be  listed  (U 
appropriate  for  the  objective,  and  if  the  number  of  items  is  not  a 
hint). 

The  directions  should  specify  whether  or  not  order  is  important.  If 
so,  the  scoring  key  should  score  sequence  separately. 

The  scoring  key  should  identify  allowable  synonyms  or  alternatives, 
and  should  specify  differential  weights  if  appropriate. 
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PERFORMANCE 


The  directions  should  clearly  explain  what  the  student  must  do  and 
how  the  item  will  be  scored. 

The  scoring  key  must  specify  all  criteria  for  all  standards  the 
performance  must  meet,  such  as  level  of  completeness,  accuracy, 
quality,  time  limit,  rate,  etc.  If  steps  in  the  performance  are 
scored,  a  checklist  should  be  provided. 

Presentation  Consistency 

i.  The  instructional  presentation  must  be  consistent  with  the  task  level  of 
the  objectives.  Certain  presentation  components  must  be  present  for  each 
task  level.  The  instructional  presentation  components  are: 

a.  STATEMENT:  The  instruction  presents  a  statement  of  a  fact,  a 
concept  definition,  the  steps  of  a  procedure  or  rule,  or  a  statement 
of  a  principle. 

b.  EXAMPLES:  The  student  is  told  or  shown  how  a  statement  of  a 
concept,  procedure,  rule,  or  principle  applies  in  a  specific  case. 

c.  PRACTICE:  The  student  practices  remembering  or  using  the 
content,  and  is  given  feedback.  PRACTICE  REMEMBERING 
components  ask  the  student  to  supply  part  or  all  of  a  fact  statement, 
concept  definition,  the  steps  of  a  procedure  or  rule,  or  the  statement 
of  a  principle.  PRACTICE  USING  components  ask  the  student  to 
use  a  concept  definition,  procedure,  rule,  or  principle  on  a  specific 
case  to  which  it  applies. 

The  REMEMBER  task  level  requires  a  statement  and  PRACTICE 
REMEMBERING.  The  USE-UNAIDED  task  level  requires  a  statement 
(or  a  review  of  the  statement),  at  least  one  example,  and  PRACTICE 
USING.  The  USE-AIDED  level  requires  examples  and  PRACTICE 
USING  with  the  aid. 

These  required  components  apply  across  all  content  types  (facts, 
concepts,  procedures,  rules,  and  principles)  for  REMEMBERING,  and 
all  except  facts  for  USING.  For  example,  if  the  objective  and  test 
item  called  for  the  student  to  remember  a  fact,  then  the  Instruction 
must  contain  a  statement  of  the  fact  to  be  remembered,  and  at 
least  one  PRAdffTCE-REMEMBERING  item  with  feedback.  No 
example  is  required,  because  it  would  be  redundant  with  the 
statement. 
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Presentation  Consistency  also  requires  that  each  required  component  meet  the  following 
criteria: 

2.  STATEMENTS  must  be  complete. 

3.  EXAMPLES  must  show  application  of  the  complete  content. 

4.  EXAMPLES  must  match  or  reflect  the  conditions  and  standards  required 

of  the  objective  and  the  test  as  closely  as  possible. 

5.  PRACTICE  must  include  feedback. 

6.  PRACTICE  must  be  of  the  same  task/content  level  as  the  test  item. 

7.  PRACTICE  must  match  or  reflect  the  conditions  and  standards  required 

of  the  objective  and  the  test  as  closely  as  possible,  or  be  designed  to 
help  the  student  gradually  learn  the  final' task. 

Most  of  the  requirements  above  are  probably  obvious,  but  some  are 
complicated.  COMPLETENESS,  for  example,  requires  different  descriptions 
for  different  content  types: 

a.  For  a  CONCEPT:  "complete”  means  that  ail  the  critical 

characteristics  of  the  concept,  and  their  combinations,  are  given. 

b.  For  a  PROCEDURE:  Complete"  means  that  all  the :  steps  of  the 
procedure  are  given  in  the  proper  order. 

c.  For  a  RULE:  “complete”  means  that  ail  the  steps  c!  the  rule  are 
given  in  the  proper  order. 

d.  For  a  PRINCIPLE:  "complete”  means  that  all  the  relevant  pre- 
and  post-conditions,  actions,  processes,  causes,  effects,  and  results 
are  stated,  and  the  relationship  between  them  is  clearly  stated. 

Presentation  Adequacy 

A  STATEMENT  is  ADEQUATE  U  it  meets  the  followir^  criteria: 

l.  The  statement  must  be  SEPARATED  from  the  rest  of  the  instruction. 
This  helps  the  student  find  the  main  idea.  When  the  statement  is  separated, 
the  key  points  stand  out,  and  arc  not  buried  in  the  presentation.  There 
are  several  ways  to  accomplish  this  goal: 

a.  Set  oft  the  statement  with  boxes. 

b.  Use  a  different  color. 

c.  Use  a  different  type,  or  underline. 

d.  Place  on  a  separate  page,  or  in  a  special  place  on  the  page, 
e*  For  audio  or  movies,  pause  before  giving  the  statement. 
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2.  The  statement  must  be  IDENTIFIED.  After  the  statement  is  separated, 
the  student  should  be  told  what  it  is.  This  permits  the  student’s  attention 
to  be  foeused  on  the  key  points  and  their  application,  rather  than  the 
student  trying  to  become  generally  familiar  with  everything  in  the 
instruction.  One  way  to  identify  a  statement  is  to  use  the  word 
’’statement.”  Other  more  content-oriented  words  are  even  more  helpful: 

definition  procedure  for  _  the  principle  of _ 

Main  Idea:  Key  Point:  General  rule: 

3.  In  addition  to  the  statement*  the  presentation  should  include  something  to 
help  the  student  better  understand  and  remember  the  statement.  Methods 
of  providing  this  help  include: 

a.  Giving  a  MNEMONIC  (memory  trick). 

b.  Giving  a  general  example  of  how  the  statement  can  be  used. 

c.  Explaining  why  the  statement  is  important. 

d.  Explaining  how  it  came  about,  how  it  fits  in  the  course,  or  how  it 

relates  to  something  the  student  already  knows. 

e.  Explaining  some  of  the  terms  in  the  statement.  : 

f.  Representing  the  statement  with  pictures,  symbols,  flowcharts, 
tables,  etc. 

ft.  Statements  for  different  content  types  must  meet  the  criteria  below: 

a.  CONCEPTS:  Give  a  decision  rule  or  search  strategy  for 

classification. 

b.  PROCEDURES:  Each  type  should  have  only  one  action. 

c.  RULES:  If  a  formula  is  used,  symbols  must  be  defined. 

EXAMPLES  are  ADEQUATE  if  they  meet  the  following  criteria: 

1.  Examples  must  be  separated  and  identified. 

2.  Examples  should  range  from  "easy”  to  "hard". 

3.  Examples  should  be  representative  of  the  job  the  student  will  do  after 

training. 

4.  There  should  be  enough  examples  to  cover  the  content  area  adequately. 

5.  Examples  should  clearly  show  why  common  errors  are  wrong. 
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The  criteria  are  generally  self-explanatory.  SEPARATED  and  IDENTIFIED  are  the  same 
as  for  statements,  and  points  2  to  5  need  no  further  explanation. 

6.  Examples  must  include  some  type  of  help. 

HELP  is  applied  in  different  ways  for  different  content  types.  Some  types  of  HELP 
for  each  content  type  are  given  below: 

a.  CONCEPTS 

Highlight  the  critical  characteristics  of  an  example. 

Explain  why  or  why  not  something  is  classified  as  a  member 
of  a  concept. 

Show  the  use  of  a  checklist  or  heuristic  to  help  classify. 

b.  PROCEDURES  or  RULES: 

Explain  why  each  step  is  done. 

Explain  why  each  step  is  important. 

Give  additional  information  about  how  to  perform  the  task. 

Give  additional  information  about  how  to  know  if  it  was 
performed  incorrectly.  - 

Give  flowcharts,  tables,  etc. 

c.  PRINCIPLES: 

Highlight  important  features. 

Simplify  the  relevant  information  from  the  case  study  in 
which  it  is  embedded. 

Use  logical  representations  of  the  if-then  relationships. 

Give  additional  information  about  why  the  principle  applies, 
or  why  it  doesn't. 

Give  hints  as  to  how  to  analyze  problems. 

PRACTICE  items  are  ADEQUATE  if  they  meet  the  following  criteria: 

1.  The  practice  section  must  be  separate  and  identified. 

2.  The  practice  items  must  be  free  of  hints  that  will  not  be  present  in  the 
test  or  on  the  job. 

3.  The  practice  items  should  have  the  same  format  as  the  format  of  the 
test  items. 


4.  The  practice  items  should  range  from  easy  to  hard. 

5.  The  practice  items  should  be  typical  of  the  job  to  be  performed  after 
traiing. 

6.  The  practice  items  should  include  the  opportunity  for  common  errors. 

7.  The  feedback  must  also  be  separated  and  identified  for  each  practice  item. 

8.  The  feedback  should  include  help  (similar  to  that  for  examples).  As  a 
bare  minimum,  the  feedback  should  direct  the  student  back  to  where  the 
instruction  was  originally  presented.  However,  it  is  better  to  have  a  new 
brief  presentation,  because  if  the  student  got  practice  wrong,-  the  original 
presentation  didh!t  help  enough. 

STATEMENTS,  EXAMPLES,  and  PRACTICE  must  be  clearly  stated.  The  following 

criteria  apply: 

1.  The  reading  level  must  be  appropriate  for  the  students. 

2.  The  presentation  should  not  be  confusing,  vague,  or  too  wordy. 

3.  All  essential  information  should  be  present;  the  student  should  not  be 
referred  to  other  places  to  obtain  information. 

4.  All  presentations  should  be  performance-oriented,  not  topic-oriented. 
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Are  the  OBJECTIVES  correctly  stated? 


a.  CONDITIONS 

o  Are  all  CONDITIONS  under  which  student  performance  is 
expected  specified? 

ENVIRONMENT:  PHYSICAL  (weather,  time  of  day,  lighting, 

etc.) 

SOCIAL  (isolation,  individual,  team,  audience, 
etc.) 

PSYCHOLOGICAL  (fatigue,  stress,  relaxed, 
etc.) 

INFORMATION:  GIVEN  INFORMATION  (scenario,  formula, 

values,  etc.) 

CUES  (signals  for  starting  or  stopping) 

SPECIAL  INSTRUCTIONS 

NON-CRITICAL  CHARACTERITICS  OF  KEY 
CONCEPTS 

RESOURCES:  JOB  AIDS  (cards,  charts,  graphs,  checklists, 

etc.) 

EQUIPMENT,  TOOLS 

TECHNICAL  MANUALS 

o  Are  the  CONDITIONS  specific  enough  to  be  unarnbig  uous? 

o  Are  the  CONDITIONS  as  close  to  real  task  CONDITIONS  as 

possible  under  existing  constraints?  (Consult  a  subject  matter 
expert.) 

o  Arc  the  CONDITIONS  which  can  be  logically  implied  deleted? 

b.  STANDARDS 

o  Are  the  STANDARDS  which  the  student  performance  must  meet 
specified? 

PERFORMANCE:  COMPLETENESS  (How  much  of  the  task  must 

uC  performed?) 

ACCURACY  (how  close  to  correct  must  the 
performance  be?) 

TIME  LIMIT  (How  much  time  is  allowed?) 

RATE  (How  fast  must  task  be  done?) 

SAFETY  (procedures  for  avoiding  hazards  in 
task  performance) 

PRODUCT:  COMPLETENESS  (What  must  finished  product 

contain?  the  precise  nature  oi  the  output) 

QUALITY  (What  objective  standard  must 
product  meet?) 

JUDGMENT  (What  subjective  opinions  must 
the  product  satisfy?) 
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•  Are  the  STANDARDS  specifically  and  unambiguously  stated  in 
measurable  terms  (trainees,  instructors,  and  evaluators  can 
discriminate  between  above-  and  below-standard  performances? 

•  STANDARDS  are  not  based  on  judgment  calls  unless  task 
outcome  can  be  classed  as  art. 

.« 

•  Are  the  STANDARDS  complete? 

•  Are  STANDARDS  for  terminal  objectives  (TLOs)  as  dose  to  real  ’ 

task  standards  as  possible?  (Consult  a  subject  matter  expert.) 

ACTION 

•  Is  the  ACTION  the  student  must  perform  specified? 

•  Is  an  action  verb  used  to  specify  what  the  student  must  do? 

•  Is  only  one  ACTION  stated  in  the  objective? 

•  Is  the  indicator  behavior  simple  enough  to  be  performed  by 
trainees? 

•  *s  t^‘e  ACTION  as  similar  to  the  real  task  as  possible? 

Section  2:  Is  the  OBJECTIVE  classifiable? 

•  Does  the  OBJECTIVE  fit  in  one  and  only  one  cell  of  the  table  in 
Figure  S?  Record  the  TASK  LEVEL  and  CONTENT  TYPE  on 
your  list  of  OBJECTIVES.  Note  that  you  should  have  read  the 
training  materials  and  done  the  practice  items  in  Appendix  E 
that  relate  to  this  section  prior  to  attempting  to  dassify  your 
OBJECTIVES. 
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3.  Is  the  OBJECTIVE  appropriate? 

•  Is  the  TASK  LEVEL  of  the  ACTION  appropriate  for  the  work  to 
be  performed  on  the  job  or  for  later  training? 

•  Is  the  CONTENT  TYPE  of  the  ACTION  appropriate  for  the  work 
to  be  performed  on  the  job  or  for  later  training? 

•  If  this  OBJECTIVE  is  REMEMBER,  is  there  a  later  USE 
OBJECTIVE? 

•  If  this  OBJECTIVE  is  USE-UNAIDED,  is  there  a  previous 
REMEMBER  OBJECTIVE?  (Not  required  if  USE-UNAIDED 
objective  is  easy  for  any  student  to  remember.) 

NOTE:  If  objectives  to  be  taught  later  are  identified  in  the  first  two  points  of 

Section  3  above,  or  there  are  OBJECTIVES  covering  a  job  aid  for  a 
USE-AIDED  OBJECTIVE  and  any  of  the  associated  objectives  are  taught 
in  the  present  course,  evaluate  those  OBJECTIVES  next  and  keep  the 
related  objectives  together  when  conducting  the  product  evaluation 
(Phase  B). 


Sample  Objectives 


In  this  section,  sample  objectives  are  rated  for  objective  adequacy.  For  each 
objective,  we  will  first  determine  whether  or  not  it  is  correctly  stated,  then 
whether  or  not  it  is  classifiable,  and  finally  whether  or  not  it  is  appropriate. 

1.  OBJECTIVE:  "The  student  will  state  the  rule  for  finding  total 

inductance  in  a  series  circuit.” 

Correctly  Conditions?  IMPLIED 

Stated? 

Standards?  IMPLIED 

In  this  objective,  our  assumptions  about  implied  conditions  and  standards 
apply.  We  are  assuming  that  the  student  will  state  in  writing  from  memory, 
with  100%  accuracy. 

Action?  O.K.  -  ONE  ACTION 

This  objective  uses  an  action  verb,  "state.”  There  is  only  one  action. 

Classifiable?  Task  Level?  REMEMBER 

Content  Type?  RULE 

Appropriate?  Conditions?  O.K.  : 

For  nearly  all  Remember-level  objectives,  the  performance  expected  is  some 
written  or  oral  response  under  fairly  normal  classroom  conditions.’ 
Occasionally,  other  conditions  might  apply,  such  as  fatigue,  noisy  environment, 
etc.  These  conditions  would  have  to  be  stated.  The  key  is  REMEMBER  THE 
JOB. 

Standards?  O.K. 

For  nearly  all  Remember -level  objectives,  the  implied  standard  of  100% 
accuracy  is  appropriate.  There  is  no  point  in  having  someone  remember 
something  unless  it  really  has  to  be  remembered.  The  only  exception  is  in  a 
familiarization  situation,  where  for  cost  reasons  the  standard  may  be  slightly 
relaxed. 


Action?  O.K. 

Later 

Use-Unaided?  ? 


This  objective  is  out  of  context.  For  a  whole  course,  we  would  expect  to  see 
a  later  Use-unaided  objective  which  requires  the  student  to  calculate  total 
inductance  in  series  circuits. 


REVISED 

OBJECTIVE: 


This  objective  is  O.K.  If  implicit  conditions  and  standards  were 
not  allowed,  the  objective  would  be:  "The  student  will  write 
from  memory  without  error  the  rule  for 


62 


2.  OBJECTIVE:  ”The  student  will  write  from  memory  the  steps  of  the 

procedure  for  field  stripping  the  SGT  YORK  cannon  as 
listed  in  the  current  edition  of  TM9-XXX-XXX-10." 

Correctly  Conditons?  IMPLIED 

Stated? 

Standards?  SPECIFIED 

Action?  O.K.  -  ONE  ACTION 

The  standard  is  "as  listed  in  the  current  edition  of  TM9-XXXX-XXX-10." 
Classifiable?  Task  Level?  REMEMBER 

Content  Type?  PROCEDURE 

Appropriate?  Conditions?  O.K. 

Standards?  O.K. 

Action?  O.K. 

Later 

Use-Unaided?  ? 

There  should  be  a  later  hands-on  objective  requiring  the  student  to  field  strip 
the  SGT  YORK  cannon. 


REVISED 

OBJECTIVE; 


O.K. 


3. 


OBJECTIVE: 


'The  student  will  describe  the  principles  of  operation  of  a 
rotary  gear  pump." 

Correctly  Conditions?  IMPLIED 

Stated? 

Standards?  INCOMPLETE 

With  Principle  objectives,  particular  care  must  be  given  to  specifying  the 
standards.  In  this  case,  the  implied  standard  is  "accurately,"  but  we  don't 
know  what  "accurately"  means.  The  solution  to  this  problem  is  to  specify 
completely  what  information  the  student's  description  must  contain.  This  can 
be  done  either  in  the  objective,  or  by  reference  to  some  other  document. 

Action?  O.K.  -  ONE  ACTION 

Classifiable?  Task  Level?  ? 

Content  Type?  PRINCIPLE 

This  objective  is  difficult  to  classify.  The  main  problem  with  the  "describe" 
action  verb  is  that  it  is  not  precise  enough  to  determine  the  task  level.  At 
this  point  in  the  objective  adequacy  procedure,  the  action  should  be  revised. 

Appropriate?  Conditions?  O.IC. 

Standards?  ? 

The  standards  should  be  revised  as  discussed  above,  and  care  should  be  taken 
to  make  sure  that  they  are  appropriate  for  the  "job."  Let's  assume  that 
recall  of  the  principle  is  required. 

Later 

Use-Unaided  ? 

If  this  objective  is  Remember,  then  there  should  be  a  later  objective  requiring 
the  student  to  use  the  principle  to  predict  something  about  the  pump's 
operation. 


REVISED 

OBJECTIVE) 


"The  student  will  write  from  memory  the  principles  of  operation 
of  a  rotary  gear  pump,  as  described  in  Tech.  Manual  XXX." 


4.  OBJECTIVE:  "Given  a  globe  valve,  rags,  prussian  blue,  gasket  material, 

packing,  tools,  and  job  aid,  the  student  will  disassemble 
and  reassemble  the  globe  valve.” 

Correctly  Conditions?  SPECIFIED 

Stated? 

Standards?  INCOMPLETE 

The  reason  the  standards  are  incomplete  is  that  there  may  be  a  time  limit 
for  performing  the  task.  If  there  is,  then  it  should  be  specified  in  the 
objective.  If  time  is  not  important,  then  the  implicit  standards  are  O.K. 

Action?  O.K. 

Although  there  are  two  action  verbs,  assemble  and  disassemble,  the  action  is 
all  right  because  the  verbs  refer  to  different  steps  in  a  procedural  task. 

Classifiable?  Task  Level?  USE-AIDED 

Content  Type?  PROCEDURE 

Appropriate?  Conditions?  O.K. 

It  is  assumed  that  all  the  materials  listed  in  the  objective  will  be  available 
on  the  job. 


Standards?  ? 

Standards  are  O.K.  if  no  time  limit  is  required. 

Action?  O.K- 

Aid 

Adequate?  ? 

If  the  job  aid  has  unfamiliar  terms  or  instructions  on  it,  there  should  be 
additional  objectives  to  clarify  the  aid. 

REVISED  This  objective  is  O.K.  unless  a  time  standard  needs  to  be 

OBJECTIVE:  specified.  A  subject-matter  expert  should  be  consulted  to  make 

this  decision. 
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5.  OBJECTIVE:  "Given  the  formula  for  Ohm’s  Law  and  two  values,  the 

student  will  solve  for  the  third  value." 

Correctly  Conditions?  INCOMPLETE 

Stated? 

The  conditions  should  specify  the  type  and  difficulty  of  the  problems  to  be 
solved.  In  addition,  because  Ohm’s  Law  can  be  written  in  several  different 
ways,  the  conditions  should  specify  which  form  or  forms  of  the  formula  will 
be  given. 

Standards?  INCOMPLETE 

The  number  of  decimal  places  should  be  specified. 

Action?  OJC.  -  ONE  ACTION 

Classifiable?  Task  Level?  USE-AIDED 

Content  Type?  RULE 

Appropriate?  Conditions?  ? 

The  type  and  difficult  of  the  problems  should  be  representative  of  the  job. 
Standards?  ? 

The  number  of  decimal  places  should  be  typical  of  what  is  required  on  the 
job. 

Action?  O.K. 

Aid 

Adequate?  ? 

The  symbols  used  in  the  formula  should  have  been  defined  previously.  This 
would  have  been  most  effectively  covered  in  a  fact-level  objective. 

REVISED  "Given  the  formula  for  Ohm's  Law,  EslR,  and  any  two  values 

OBJECTIVE:  typical  of  the  values  encountered  on  the  job,  the  student  will 

solve  for  the  third  value  correct  to  two  decimal  places.” 


6. 


OBJECTIVE: 


Correctly 

Stated? 


Classifiable? 


Appropriate? 


"Given  schematic  symbols  for  common  electronic 
components,  the  student  will  write  from  memory  the 
name  of  the  component  represented  by  each  symbol.” 


Conditions? 

Standards? 

Action? 

Task  Level? 
Content  Type? 
Conditions? 


SPECIFIED 

IMPLIED 

O.K.  -  ONE  ACTION 

REMEMBER 

FACT 

O.K. 


We  will  assume  that  the  "common  electronic  components"  are  ones  that  are 
typical  of  the  equipment  the  student  will  be  trained  to  maintain  or  repair.  If 
there  were  special  components  or  symbols  that  the  student  had  to  know,  then 
these  should  be  included. 

Standards?  O.K. 


Action?  O.K. 

Later 

Use-Unaided?  ? 


There  should  be  a  later  Use  objective  for  which  the  student  must  be  able  to 
recall  component  names  given  component  symbols. 

REVISED  O.K. 

OBJECTIVE: 
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7.  OBJECTIVE:  "Given  the  explanation  of  the  principle  of  supply  of  a 

military  force  from  the  text,  the  student  will  describe  how 
this  principle  applies  in  Hitler's  attack  on  Russia,  the 
Battle  of  Midway,  the  Battle  of  the  Bulge,  and  Sherman's 
march  through  Georgia." 

Correctly  Conditions?  SPECIFIED 

Stated? 

Standards?  INCOMPLETE 

The  implied  standard  is  "correctly"  but  with  principle  objectives  it  is  often 
difficult  to  tell  what  correctly  means.  Therefore,  either  the  correct 
explanation  should  be  included  in  the  objective,  or  a  document  containing  the 
correct  explanation  should  be  referenced. 

Action?  O.Y  -  ONE  ACTION 

Classifiable?  Task  Level?  USE-AIDED 

Content  Type?  PRINCIPLE 

Appropriate?  Conditions?  ? 

Standards?  See  above 

Action?  ?  * 

The  appropriateness  questions  depend  on  how  the  student  is  going  to  use  the 
principle  on  the  job.  If  the  Job  is  to  plan  military  operation  given  reference 
materials,  then  the  conditions  are  probably  appropriate.  If,  however,  the  job 
involves  making  quick  decisions  on  the  battle  field,  then  reference  materials 
would  not  be  appropriate,  and  the  task  level  should  then  be  Use-Unaided. 

The  action  of  discussing  how  the  principle  of  supply  applied  in  previous 
battles  may  or  may  not  be  appropriate,  depending  on  whether  or  not  the 
principle  still  holds  in  modern  warfare. 

Aid 

Adequate?  ? 

Is  the  explanation  given  in  the  text  sufficient  to  allow  the  student  to  give  a 
correct  discussion,  or  should  other  objectives  on  the  a^d  be  included? 

REVISED  The  standard  should  be  revised  as  indicated  above.  Other 

OBJECTIVE:  revision  would  depend  on  answers  to  the  appropriateness  questions 

we  have  raised. 


8. 


OBJECTIVE: 


"The  student  will  solve  for  inductive  reactance  in  a 
circuit,  given  frequency  and  inductance." 

Correctly  Conditions?  UNCLEAR 

Stated? 

Is  the  student  given  the  circuit,  or  just  the  values  for  frequency  and 
inductance?  If  he  is  given  the  circuit,  is  it  the  actual  equipment,  or  a 
schematic? 

Standards?  IMPLIED 

Action?  O.K.  -  ONE  ACTION 

Classifiable?  Task  Level?  USE-UNAIDED 

Content  Type?  RULE 

Appropriate?  Conditions?  ? 

The  revised  conditions  should  be  appropriate  for  the  job. 

Standards?  O.K. 

Action?  O.K. 

Previous 

Remember?  ? 

There  should  be  a  previous  objective  requiring  the  student  to  recall  the 
formula  for  solving  for  Inductive  reactance. 

REVISED  "Given  a  circuit  schematic  with  values  of  frequency  and 

OBJECTIVE:  inductance  specified,  the  student  will  solve  for  inductive 

reactance." 
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9. 


OBJECTIVE: 


,rThe  student  will  perform  the  steps  required  to  accept, 
verify,  and  log  messages  to  be  transmitted  via  teletype 
tape.” 

Correctly  Conditions?  INCOMPLETE 

Stated? 

The  conditions  should  specify  the  equipment  the  student  will  use  and  the 
environmental  conditions. 

Standards?  INCOMPLETE 

A  time  or  rate  standard  may  be  required.  A  subject-matter  expert  should  be 
consulted. 

Action?  ?  -  MULTIPLE  ACTION 

If  accept,  verify,  and  log  are  all  steps  of  a  procedure  for  transmitting 
messages,  then  the  action  is  O.K.  If,  however,  cne  or  more  of  the  action 
verbs  involves  a  different  content  type,  then  additional  objectives  should  be 
written. 

Classifiable?  Task  ,evel?  U5E-UNAIDED 

Content  Type?  PROCEDURE 

Appropriate?  Conditions?  ? 

Standards?  ? 

Action?  ? 

Any  revisions  made  to  the  conditions,  standards,  and  action  should  be 
representative  of  the  job. 

Previous 

Remember?  ? 

There  should  be  a  previous  remember  objective  that  requires  the  student  to 
memorize  the  steps  of  the  procedure. 

"In  a  classroom  laboratory  simulating  a  radio  room,  the  studer 
will  perform  the  steps  required  to  process  (accept,  verify,  log) 
messages  to  be  transmitted  via  teletype  tape,  at  the  rate  of 
5  minutes  per  message." 


REVISED 

OBJECTIVE: 


10.  OBJECTIVE: 


•The  student  will  use  the  principle  of  electromagnetic 
induction  to  describe  the  operation  of  an  AC  generator,” 


Correctly  Conditions: 

Stated? 

Standards: 

The  standard  must  specify  what 
order  to  be  correct. 

Action? 

Classified?  Task  Level? 

Content  Type? 

Appropriate?  Conditions? 

Standards? 

Action? 


IMPLIED 

INCOMPLETE 

the  student's  description  must  contain  in 

O.K.  -  ONE  ACTION 

USE-UNAIDED 

PRINCIPLE 

O.K. 

See  Above. 

? 


On  the  job,  will  the  student  ever  have  to  use  his  knowledge  of 
electromagnetic  induction  to  describe  the  operation  of  equipment?  If  so,  then 
this  objective  is  appropriate.  On  the  other  hand,  if  the  intent  was  just  to 
have  the  student  learn  the  principles  of  operation  of  the  AC  generator,  then 
the  objective  should  be  to  "Remember"  the  principles  of  operation.  As  it 
stands,  the  objective  might  be  appropriate  if  the  student  will  be  required  to 
troubleshoot  new  AC  generators. 

Previous 

.  Remember?  ? 


There  should  be  an  earlier  objective  requiring  the  student  to  remember  the 
principle  of  electromagnetic  induction. 

REVISED  "The  student  will  use  the  principle  of  electromagnetic 

OBJECTIVE:  induction  to  describe  the  operation  of  an  AC  generator.  The 

description  must  contain  the  following  points..." 
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11.  OBJECTIVE: 


"Given  the  guidelines  for  determining  message  security 
classification,  the  student  will  determine  the  security 
classification  (Top  Secret,  Secret,  Confidential,  or 
Unclassified)  for  outgoing  messages." 


Correctly 

Conditions? 

SPECIFIED 

Stated? 

Standards? 

IMPLIED 

Action? 

O.K.  -  ONE  ACTION 

Classifiable? 

Task  Level? 

USE-AIDED 

Content  Type? 

CONCEPT 

Appropriate? 

Conditions? 

O.K. 

Standards? 

O.K. 

Action? 

O.K. 

Aid 

Adequate? 

? 

The  guidelines  should  be  checked  to  make  sure  they  are  complete,  and  are 
detailed  enough  so  that  the  student  can  use  them  to  perform  the  task. 

REVISED  This  objective  is  probably  O.K.  if  the  aid  is  adequate. 

OBJECTIVE: 
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4. 


Is  the  coverage  of  the  LOs  complete?  (TLOs  have  already  been  considered  in 
Step  A4.1.) 


o  Does  a  list  of  LO  actions  or  an  objectives  hierarchy  exist  for 
each  TLO? 

o  Whenever  any  test  item  or  training  event  addreses  a  unit  of 

instruction  smaller  than  a  TLO  and  you  have  decided  to  conduct 
the  TEE  at  that  level,  does  a  list  of  LO  actions  or  an  objectives 
hierarchy  exist  for  each  TLO? 

o  Are  complete  objectives  written  for  each  critical  element  of  the 
task?  Label  LOs  as  critical  which  have  the  following 
characteristics: 

They  are  some  of  the  most  measurable  and  observable 
elements  of  the  task. 

They  have  serious  consequences  of  inadequate  performance. 

They  are  common  sources  of  failure. 

They  are  actions  which  the  soldiers  of  the  target 
population  do  not  already  know  how  to  do  and  must 
therefore  be  taught. 

NOTE:  When  evaluating  LOs,  note  that  a  common  type  of  LO  for  procedure 

and  rule  tasks  in  equipment-related  MOS  is  a  "Remember  LO"  on  the 
location  of  certain  controls  or  pieces  of  equipment.  Along  with 
another  objective  requiring  the  recall  of  the  related  steps  in  the 
procedure  or  rule,  an  LO  on  equipment  location  would  be  subordinate  to 
a  higher  LO  or  TLO.  That  LO  or  TLO  would  require  the  performance 
of  the  steps.  For  example: 


■V 

s 


Note,  however,  that  "Re/nember  LOs"  will  have  indicator  behaviors 
different  front  the  kind  shown.  (Remember  objectives  are  taught 
before  their  corresponding  skill  objectives  at  the  use  level  are  trained.) 
As  shown  in  the  diagram  above  they  are  unobservable  as  independent 
skills,  but  are  measured  by  watching  the  performance  of  the 
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superordinate  LO.  At  the  remember  level  the  controls  would  have  to 
be  "pointed  out"  on  the  equipment  or  a  picture  of  it  and  the  procedure 
would  have  to  be  stated  in  some  way,  rather  than  the  procedure  being 
performed.  This  level  of  detail  will  be  useful  only  when  such 
remember -level  instruction  exists  for  a  complex  task. 

•  Are  subordinate  OBJECTIVES  common  to  more  than  one  LO  or 
TLO  so  identified? 


Task  A5:  Specify  training  event  types  and  select  events  for  observation. 

A5.1  Using  the  documentation  collected  in  previous  Phase  A  tasks  and  your 

knowledge  of  the  course,  make  a  list  of  the  types  of  training  events 
that  are  employed.  A  list  of  some  possible  kinds  of  events  is  given 
below.  You  should  label  the  types  of  events  in  your  course  and  define 
them  if  they  are  different  from  the  definitions  shown  below.  Consult 
one  of  the  course  instructors  if  you  need  help.  (Note  that  your 
materials  may  be  such  that  it  would  be  more  efficient  to  discover  the 
training  event  types  as  you  do  Step  A3. 2) 


A  lecture  or  discussion  involving  a  group  of 
students  and  art  instructor.  Statements,  state¬ 
ment  help,  examples,  and  example  help  are 
the  most  commonly  used  presentation  compo¬ 
nents  in  the  classroom;  however,  practice, 
feedback,  and  feedback  help  could  possibly  be 
employed  for  some  non-equipment  related 
tasks  or  for  remember  level  objectives. 

DEMONSTRATION:  Showing  the  students  when  and  how  to  per¬ 

form  a  procedure  in  a  realistic  setting  such 
as  on  real  equipment.  A  demonstration  is  a 
realistic  example  and  can  employ  example 
help.  Most  often  it  also  includes  a  statement 
of  the  procedure  given  with  the  example. 

PRACTICE:  Allowing  trainees  to  perform  one  task  isolated 

from  the  performance  of  other  tasks.  This 
could  apply  to  individuals,  groups,  or  teams, 
although  team  practice  is  more  often  of  the 
integrated  type  defined  below.  Practice, 
feedback,  and  feedback  help  are  the  related 
presentation  components.  Practice¬ 
remembering  and  practice-using  could  be  desig¬ 
nated  as  separate  events  although  practice- 
remembering  would  usually  be  included  in 
classroom  instruction  or  individual  study. 

PERFORMANCE  TEST:  Assessment  of  how  well  students  can  perform 

tasks  at  the  use  task  levels  in  isolation  from 
other  tasks. 


CLASSROOM 

INSTRUCTION: 
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WRITTEN  TEST: 


Assessment  of  how  well  students  can  perform 
at  any  task  level  using  paper  and  pencil  tests. 


A5.2 


A5.3 


A5  A 


ORAL  TEST:  Assessment  of  how  well  students  can  perform 

at  any  task  level  using  spoken  responses  by 
the  students. 


INTEGRATED 

PRACTICE  OR  TEST:  Practice  or  testing  of  two  or  more  tasks  in 

conjunction  with  each  other  at  high  level  of 
realism  or  in  an  operational  setting.  This 
may  be  done  by  an  individual,  a  group 
performing  individual  tasks,  or  a  team 
performing  team  functions. 

INDIVIDUAL  STUDY:  A  situation  in  which  students  are  allowed  to 

study  on  their  own.  It  could  follow  classroom 
instruction  or  it  could  occur  in  individualized 
instructional  courses  where  students  typically 
proceed  at  their  own  pace. 

HELP  SESSION:  Instruction  of  any  of  a  number  of  the  types 

held  in  addition  to  the  regular  course 
instruction  to  aid  students  who  need 
additional  help.  Statement  or  example  help 
as  well  as  practice  and  feedback  would  be 
the  typical  presentation  components  employed. 

You  will  observe  training  events  and  evaluate  the  corresponding  presen¬ 
tation  materials  and  tests  for  each  task  selected  for  evaluation  in  Task 
A3.  The  next  three  steps  identify  the  events  that  apply  to  the  tasks 
which  were  selected.  You  may  need  an  instructor’s  help  at  some  point. 

Locate  documentation  which  outlines  the  lessons  and  tests  to  be  given 
in  the  course,  such  as  the  Program  of  Instruction  (POI)  and/or  lesson 
plans  and  tests.  If  the  documentation  does  not  provide  a  dear  outline 
showing  the  lesson  topics  broken  down  into  the  training  event  types 
identified  in  Step  A5.1,  either  amend  your  outline  accordingly  or  list 
the  topics  with  the  events  in  a  separate  list. 

Create  a  column  (to  the  left  or  the  right  of  your  outline)  labelled 
"TASKS.”  Refer  to  Worksheet  A3  and  any  course  materials  you  may 
need.  List  the  task  number  for  each  task  not  eliminated  on  Worksheet 
A3  by  each  lesson  of  which  it  is  a  part.  It  is  a  part  of  the  lesson  if 
it  is  taught  or  tested  in  the  lesson  and  must  be  performed  or 
understood  in  order  for  other  tasks  in  the  lesson  to  be  learned. 


Cross  off  the  lessons  with  no  task  numbers  beside  them.  Next, 
examine  all  of  the  training  events  listed  for  the  rc*n»aining  lessons.  If 
any  event  applies  only  to  a  task  or  tasks  which  are  not  listed  for  the 
lesson,  cross  it  off.  If  an  event  covers  both  tasks  selected  for 
evaluation  and  any  that  were  not,  mark  the  event  with  an  astciisk  (*) 
and  write  this  footnote  at  the  bottom  of  the  page:  "*  events  for 
which  only  some  tasks  are  to  be  evaluated." 
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PHASE  B:  CONDUCT  PRODUCT  EVALUATION 


Overview  of  Phase  B 


TASKS: 

Bl.  Select  product  evaluation 
questions 


B2.  Evaluate  test  materia  Is 


B3.  Evaluate,  presentation 
materials 


RELEVANT  WORKSHEET: 

Master  List  of  Evaluation 
Questions  (Appendix  A)  and 
Product  Evaluation  3PA 
(Appendix  C) 

B2  (Appendix  H),  Master  List  of 
Evaluation  Questions  //I  -  26 
fAppendix  A),  and  Product 
Evaluation  3PA  (Appendix  C) 

B3  (Appendix  H),  Master  List  of 
Evaluation  Questions  #2 7  -  94 
(Appendix  A),  and  Product 
Evaluation  3PA  (Appendix  C) 


Guidelines  for  Phase  B 

Task  Bl:  Select  product  evaluation  questions. 

In  this  task  the  Master  List  of  Evaluation  Questions  (Appendix  A)  has 
its  first  use.  It  is  divided  into  two  sections:  one  for  tests  (used  in 
Task  B2),  and  one  for  the  presentation  of  the  instruction  (used  in  Task 
B3).  In  this  task  you  will  identify  those  questions  that  are  applicable 
to  a  product  evaluation  (conducted  using  the  course  materials).  Also, 
you  will  limit  the  evaluation  to  those  applicable  to  the  training 
situation.  A  similar  procedure  is  followed  for  process  evaluation  in 
Task  Cl. 

Make  a  copy  of  the  job  performance  aid  (3PA)  for  product  evaluation 
in  Appendix  C  and  turn  to  the  Master  List  of  Evaluation  Questions  in 
Appendix  A.  Using  the  copy  of  the  JPA  as  your  recording  instrument, 
examine  each  question  listed  in  the  Master  List  and  cross  out  (on  the 
3PA)  the  ;e  questions  that  do  not  apply  to  your  TEE  for  any  of  the 
following  reasons: 

t  Several  questions  contain  an  “if  clause”  restricting  their  use  to 
certain  types  of  subject  matter  or  to  certain  conditions.  Do  not 
include  questions  inappropriate  to  the  situation. 

•  To  what  degree  are  the  course  presentation  and  test  dependent 
on  an  instructor  or  on  print  or  other  types  of  mediated  course 
materials?  This  is  the  key  question  in  deciding  whether  product 
evaluation  is  possible  for  a  given  TEE  question.  If  not,  do  not 
include  the  questions  here. 
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•  Some  questions  are  labelled  *PRODUCT  ONLY  in  the  Master  List 
and  should  always  be  answered.  They  are  labelled  "PRODUCT 
ONLY"  because  they  may  be  too  difficult  for  a  data  collector, 

as  defined  in  the  introduction,  to  answer.  The  asterisk  (*)  on 
the  "PRODUCT  ONLY"  designation  distinguishes  these  questions 
from  other  PRODUCT  ONLY  questions  in  the  Master  List.  If 
any  of  the  designated  questions  cannot  be  answered  in  a  product 
evaluation  and  they  apply  to  the  TEE,  the  TEE  analyst  should 
make  arrangements  to  answer  them  in  the  process  evaluation. 

Use  the  questions  remaining  on  your  3PA  to  conduct  the  product 
evaluation  in  Tasks  B2  and  B3.  Note  that  if  there  is  more  than  one 
type  of  test  used  in  the  course,  e.g.,  written,  oral,  and  performance, 
you  should  label  each  question  with  the  test  type(s)  that  apply. 

Task  B2:  Evaluate  test  materials* 

B2.1  Bring  together  the  course  and  TEE  documentation  you  will  need  to 

conduct  this  task: 

•  The  course  objectives 

•  All  tests  associated  with  the  course 

•  All  test  administrative  directions 

•  Materials  used  for  scoring  the  tests,  such  as  keys  or  manuals 
which  contain  standards  of  performance  for  certain  items 

•  Your  course  outline  from  Task  A 5 

•  Your  edited  copy  of  the  product  evaluation  3PA 

•  The  Master  List  of  Evaluation  Questions  (Appendix  A) 

B2.2  Find  your  list  of  objectives  (or  tasks).  You  should  have  already 

classified  each  one  by  task  level  and  content  type  in  Task  A4.  If  you 
have  LOs,  both  TLOs  and  LOs  should  have  been  classified. 

Referring  to  both  your  objective  list  and  the  test  or  group  of  tests 
that  constitute  the  course  final  exam,  locate  the  test  questions  that 
test  each  objective  and  write  them  on  your  list  by  the  appropriate 
objectives.  As  you  do  so,  classify  each  test  question  by  task  level  and 
content  type.  Record  those  classifications  along  with  the  test  question 
numbers. 

B2.3  Using  the  lesson  titles  on  your  course  outline  as  a  general  guide,  and 

other  course  materials  as  needed,  take  copies  of  the  tests  and  cross 
out  the  items  or  item  numbers  of  the  items  that  do  not  apply  to  tasks 
selected  for  evaluation.  In  Step  B2.4,  evaluate  only  the  items  which 
are  not  crossed  off. 
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Guidelines  for  Worksheet  B2 


B2.4  Locate  Worksheet  B2  in  Appendix  H.  Based  on  the  number  of  test 

items  you  have  to  evaluate,  make  enough  copies  of  Worksheet  B2  to 
conduct  the  evaluation.  You  may  wish  to  copy  Worksheet  B3 
(Appendix  H)  at  this  time  as  well. 

Conduct  the  evaluation  by  asking  each  question  remaining  on  your  3PA 
beginning  with  Question  1  and  proceeding  through  Question  26.  Record 
your  ratings  on  Worksheet  B2.  As  is  also  directed  on  the  JPA, 
organize  your  work  according  to  the  following  conventions:  . 

•  If  you  evaluate  more  than  one  test  (i.e.,  entry  tests,  pretests, 
lesson  tests,  or  encf«of~course  tests),  ask  all  questions  for  each 
test  before  evaluating  the  next  test. 

•  Questions  1  through  3  should  be  asked  of  each  objective  before 
going  on  to  the  next  objective. 

•  Questions  4  through  18  should  be  asked  for  each  test  item  before 
continuing  on  to  the  next  item.  Also  the  items  which  apply  to 
each  objective  should  be  covered  before  going  on  to  the  next 
objective.  Therefore,  record  objective  numbers  consecutively 
(TLOs,  LOs,  or  both  —  LOs  within  TLOs)  and  the  test  items 
which  apply  to  each  one  for  those  questions. 

•  Questions  19  through  26  should  be  asked  once  for  an  entire  test. 
The  word  "TEST”  may  entered  in  the  'TTEM  it"  column  of 
Worksheet  B1  to  so  indicate. 

The  Master  List  of  Evaluation  Questions  (Appendix  A)  gives  much  more 
specific  guidance  for  most  questions  than  the  product  JPA.  You  should 
be  sure  you  understand  each  question  before  relying  exclusively  on  the 
3PA. 


Be  sure  to  record  the  Task/TLO//,  Lesson  Item  //,  and  TEE  Question 
0  accurately.  Otherwise,  you  will  have  difficulty  interpreting  the  data 
in  your  analysis  during  Phase  E. 

In  the  Description  column  of  your  worksheet,  supply  a  brief  description 
for  every  problem  rated  which  would  not  be  self-explanatory  by 
referring  back  to  the  evaluation  question  in  the  Master  List. 

Task  B3:  Evaluate  presentation  materials. 


It  makes  little  difference  when  you  conduct  this  task,  as  long  as  it  is 
done  before  Task  E2:  Summarize  Product  and  Process  Evaluation  Data. 
Depending  on  the  purpose  of  your  TEE,  which  you  and  your  sponsor 
selected  in  Task  A 1 ,  you  may  wish  to  postpone  this  task  (B3)  until 
after  performance  scores  are  collected  and  analyzed  in  Phase  E.  You 
may  then  choose  to  conduct  this  task  (and  corresponding  steps  in  Phase 
E)  only  for  those  tasks  taught  in  the  course  which  are  found  to  have 
perfor  mance  discrepancies.  Indeed,  it  is  not  the  best  use  of  resources 
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to  conduct  this  task  for  all  tasks  selected  for  evaluation,  unless  it  is 
already  known  that  the  course  is  functioning  quite  poorly  on  most  of 
the  tasks  taught.  If  you  choose  this  route,  conduct  the  process 
evaluation  as  usual,  but  analyze  only  the  discrepant  tasks  on  which  you 
conduct  the  product  evaluation  when  you  analyze  the  presentation  data. 

B3.1  Gather  the  course  documentation  you  will  need  to  conduct  this  task: 

•  The  course  objective 

•  All  lesson  materials  and  manuals  used  by  the  students 

•  Any  A/V  equipment  needed  to  hear  or  view  the  materials 

•  Lesson  and  course  administrative  directions 
Guidelines  for  Worksheet  B3 

B3.2  Locate  Worksheet  B3  in  Appendix  H  if  you  have  not  already  done  so, 

and  make  enough  copies  to  conduct  the  evaluation. 

The  amended  product  evaluation  3PA  produced  in  Task  B1  and  the 
course  outline  with  events  to  be  evaluated  from  Task  A5  are  also 
required  for  this  task. 

B3.3  Conduct  the  evaluation  by  asking  each  question  you  specified  on  your 

3PA  starting  with  Question  27  and  proceeding  thrugh  Question  93. 
Record  each  rating  on  Worksheet  B3.  In  general,  ask  the  questions 
once  for  each  lesson  which  is  included  in  the  TEE.  Ask  each  question 
which  does  not  apply  to  whole  lessons  for  each  objective  within  the 
lesson  before  going  to  the  next  question.  Whether  you  use  TLOs  alone 
or  LOs  grouped  within  TLOs  depends  on  decisions  made  in  Task  A4. 

Use  only  the  objectives  that  apply  to  training  events  selected  in  Task 
A5. 

Many  questions  address  presentation  components  within  objectives,  such 
as  examples  or  practice.  Ask  these  questions  in  groups  as  they  appear 
in  your  3PA  for  each  LO  or  TLO  before  going  on  to  the  questions  at 
the  TLO  level. 

Also,  a  few  questions  apply  to  the  course  as  a  whole,  or  at  only  one 
point  in  the  course  (c.g.,  Question  29).  Use  these  questions  only  once 
for  the  entire  course  after  you  have  asked  the  prccceding  questions  of 
each  lesson,  TLO,  LO,  and  presentation  component. 

Be  sure  to  record  the  lesson  number  or  title,  task/TLO //,  LO//,  or  LO 
action,  the  type  of  presentation  component,  and  the  TEE  Question 
number  accurately.  Again,  you  will  have  difficulty  interpreting  the 
data  during  Phase  E  if  you  do  not.  Numbers  and  letters  need  be 
recorded  only  when  they  change. 

In  the  Description  column  of  your  worksheet,  supply  a  brief  description 
for  every  problem  rated  which  would  not  be  self-explanatory  by 
referring  back  to  the  evaluation  question  in  the  Master  List. 


PHASE  Cs  PLAN  TRAINING  PROCESS  EVALUATION 
Overview  of  Phase  C 


TASKS: 

Cl.  Prepare  process  evaluation 
worksheets. 

C2.  Make  logistical  arrangements 
to  conduct  the  training 
process  TEE. 


RELEVANT  WORKSHEET: 

Cl.l  through  Cl.  15  (Appendix  H) 

C2  (Appendix  H) 


Guidelines  for  Phase  C 

Task  Cl:  Prepare  process  evaluation  worksheets. 

Guidelines  for  Worksheets  Cl.l  to  C1.8 

Cl.l  Locate  your  list  of  training  event  types  identified  in  Task  A 5  and  turn 

to  Worksheets  Cl.l  through  0.8  in  Appendix  H.  Make  copies  of  those 
worksheets  that  apply  to  each  of  the  training  event  types  which  you 
have  identified.  Note  that  a  help  session  can  be  either  classroom 
instruction,  demonstration,  practice,  integrated  practice,  or  .a 
combination  of  these.  There  is  no  separate  worksheet  for  help  sessions. 

Ci.2  A  general  heading  is  shown  on  each  worksheet  which  should  be 

applicable  to  most  training  settings.  You  may  wish  to  adapt  the 
heading  to  your  own  training  situation,  however,  following  the  rules 
below.  The  first  set  of  points  listed  below  is  essential  and  should 
always  be  included; 

o  Date 

o  Data  collector 

o  Title,  giving  the  name  of  the  training  event,  and  course  title 
o  TLO/lesson  P  and/or  title 

o  Class  P  or  MOS  P  (if  there  is  more  than  one  class  or  MOS) 

o  Instructor  code  to  identify  the  instructor 

o  Starting  time 

o  Training  situation  or  environment 

o  Whether  practice  is  for  individuals,  crews,  or  larger  units,  if 
applicable 


These  points  may  be  included: 
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•  LO#,  for  complex  tasks  with  training  events  directed  at  LOs  or 
groups  of  LOs  smaller  than  a  task 

•  Ending  time 

•  //  of  trainees  involved 

In  addition,  worksheet  headings  for  tests  should  include  the  following 
points; 

•  Whether  the  test  is  for  an  individual,  a  crew,  or  a  larger  unit 
(as  many  of  these  as  may  apply) 

•  Who  is  scoring  the  test 

In  general,  you  should  not  include  information  that  would  not  be  useful 
in  identifying  the  worksheet  at  a  later  time  or  that  would  not  be 
useful  in  analyzing  the  training  data. 

You  should  enter  items  which  will  always  be  the  same  on  the  master 
copy  of  each  worksheet  (before  reproduction).  The  course  title  or 
number  is  one  such  item.  Also,  alternate  group  (class)  and  MOS 
numbers  can  be  entered,  if  applicable,  and  the  instruction  "circle" 
included  above  them. 

Referring  to  the  Master  List  in  Appendix  A,  go  through  the  questions 
on  each  worksheet  you  have  selected  for  the  training  events  in  your 
TEE  and  cross  out  those  that  are  not  applicable.  Consider  the 
following  points  in  deciding  which  questions  to  eliminate: 

•  Is  the  subject  matter  such  that  for  any  given  question  each  part 
of  the  instruction  would  be  rated  as  adequate?  That  is,  from 
what  you  know  is  it  likely  that  there  would  every  be  anything 
wrong  with  any  part  of  the  instructon  for  that  question?  If  not, 
the  question  should  not  be  included  in  the  TEE. 

•  Several  questions  contain  an  "if  clause"  restricting  their  use  to 
certain  types  of  subject  matter  or  to  certain  conditions.  Cross 
out  questions  that  do  not  apply  for  these  reasons. 

e  Are  the  training  and/or  course  materials  accessible  enough  so 

that  the  question  can  be  answered  satisfactorily  from  the  observa¬ 
tions  possible  in  the  training  environment?  Occasionally  there 
are  tasks  which  must  be  practiced  or  tested  in  environments 
inaccessible  to  an  outside  observer,  such  as  the  inside  of  a  tank 
or  an  aircraft  in  flight  with  a  limited  number  of  crew  positions. 

•  If  the  point  in  a  question  cannot  be  observed,  ascertain  whether 
you  can  gain  the  information  in  some  other  way.  For  example, 
you  may  be  able  to  listen  to  the  training  in  some  way  or  you 
might  be  able  to  get  instructor  or  trainee  reactions  on  wliat 
Iwppened  that  would  be  valid  for  some  questions.  (If  so,  make  a 
list  of  alt  such  questions  for  use  in  Step  C1.6.)  How  important 
is  the  question  with  respect  to  the  effort  that  will  be  required 
to  get  the  data? 
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NOTE:  If  there  are  large  number  of  questions  which  cannot  be  observed,  you 

may  wish  to  record  their  numbers  as  well,  and  make  an  assessment  in 
your  report  in  Phase  F  of  their  effect  on  the  validity  of  the  TEE. 

•  To  what  degree  are  the  course  presentation  and  test  dependent 
on  an  instructor  or  on  print  or  mediated  course  materials?  If 
the  question  can  be  answered  only  by  examining  course  materials, 
it  should  not  be  included  here,  but  should  be  included  in  a 
product  evaluation  in  Phase  B. 

Once  you  have  marked  the  questions  to  be  eliminated,  you. can  give 
your  worksheets  a  less  cluttered  appearance.  Either  black  out  the 
inapplicable  questions  or  cut  them  out  and  tape  the  remaining  questions 
back  together  (observing  an  appropriate  page  length),  prior  to  copying 
the  worsheets  for  use  in  data  collection. 

Guidelines  for  Worksheet  C1.9 

Worksheet  C1.9  in  Appendix  H  is  a  special  worksheet  that  applies  to 
Master  List  Questions  32,  73,  and  76.  As  the  TEE  analyst,  you  will  be 
the  only  one  to  use  it,  since  it  is  filled  out  by  reviewing  process  data 
collection  worksheets  from  a  series  of  training  events.  It  should  be 
used  during  Task  D2  or  sometime  prior  to  conducting  Task  E3. 

NOTE:  The  steps  remaining  in  this  task  concern  the  collection  of  trainee  and 

instructor  reaction  data.  The  relative  importance  of  trainee 
reactions/opinions  is  somewhat  controversial  in  the  evaluation  field. 
Trainee  opinions  are  the  only  data  collected  by  some  evaluators.  Other 
evaluators  give  it  so  little  weight  that  they  do  not  collect  it  at  all. 

The  authors  of  your  guidelines  take  the  position  that  trainees  arc  not 
particularly  qualified  to  make  subjective  judgments  about  the  quality  of 
training.  They  are,  however,  quite  capable  of  observing  what  happened 
in  the  training  and  stating  how  it  affected  either  the  way  they  learned 
the  materiat  cr  their  motivation  for  learning  it.  Therefore,  trainee 
reactions  which  pinpoint  individual  learnir^  problems  or  points  in  the 
instruction  that  substantially  reduced  motivation  for  learning  are 
appropriate  for  TEE  data  collection.  Also,  when  circumstances  prohibit 
other  observers  from  seeing  or  hearing  the  instruction,  trainees  should 
be  asked  what  happened.  Evaluators  can  then  "rate"  the  instruction 
appropriately. 

Except  when  trainee  observations  are  the  only  data  available,  it  is 
assumed  that  trainee  reactions  will  be  used  as  a  supplement  to  other 
evidence  in  analyzing  training  discrepancies  in  Phase  E. 

In  some  TEE  situations,  you  may  not  be  able  to  collect  trainee  opinions 
at  all  because  of  policy.  In  others,  you  may  not  be  able  to  use 
instruments  from  this  guide  but  will  have  to  rely  on  instruments 
prepared  by  other  organizations.  In  some  situations,  you  may  not  be 
able  to  interview  samples  of  the  trainees. 

Thus,  there  seem  to  be  four  alternatives  regarding  the  collection  of 
trainee  opinions  as  part  of  the  TEE: 
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•  No  data  at  all  can  collected  from  trainees. 

•  Interviews  of  trainees  can  be  conducted. 

•  Instruments  prepared  by  other  organizations  can  be  used. 

•  Data  can  be  collected  from  instruments  you  prepare  following 
the  guidelines  in  this  document. 

Cl. 4  If  you  prepare  the  instruments  described  in  the  next  few  steps,  contact 

the  appropriate  commands  and  insure  that  you  will  be  able  to 
implement  reaction  questionnaires  and  interview  samples  of  the  trainees 
and  instructors*  as  indicated. 

Guidelines  for  Worksheets  C1.10  and  CL  11 

C1.5  Look  at  Worksheets  CLIO  and  Cl.ll,  the  trainee  reaction  instruments. 

They  contain  the  same  basic  questions,  but  have  different  headings  and 
are  structured  so  they  can  be  used  as  either  a  written  questionnaire 
(CLIO)  or  an  interview  guide  (Cl.ll).  If  you  have  time  and  the 
cooperation  of  the  training  organization,  the  collection  of  trainee 
reaction  data  can  provide  a  useful  supplement  to  the  data  collectors' 
observations.  In  some  cases,  the  only  way  to  observe  certain  aspects 
of  the  training  will  be  to  ask  trainees  (or  instructors)  what  happened  in 
the  training  using  Master  List  questions  (see  Step  C1.6.)  In  any  case, 
trainee  reaction  data  is  likely  to  be  useful  only  if  the  trainees  have 
been  assured  that  their  responses  will  not  be  revealed  to  their 
instructors! 

In  this  step  make  a  final  decision  on  whether  or  not  to  collect  trainee 
reaction  data  and  if  so,  which  method  to  use,  either  questionnaire  or 
interview.  Larger  samples  of  trainees  can  potentially  be  employed  with 
written  questionnaires,  but  in  interviews  you  can  ask  about  whatever 
details  you  feel  are  the  most  important. 

The  questions  shown  on  Worksheets  CLIO  and  Cl.ll  deal  with  the 
student's  perceptions  of  what  happened  in  training  and  how  it  affected 
his  learning  or  motivation  to  iearru 

CL6  For  each  situation  which  you  cannot  observe  directly,  provide  a  special 

worksheet.  Use  a  copy  of  each  worksheet  you  have  amended  in  Step 
CL3  which  would  normally  be  used  to  observe  that  situation.  Simply 
Include  the  words  "TRAINEE  INTERVIEW  FORM"  at  the  top  of  the 
workshop  before  reproducing  it.  Note  that  only  the  TEE  analyst  or  a 
well-trained  associate  analyst  will  be  used  to  give  these  interviews, 
since  each  question  must  be  reworded  as  it  is  being  asked. 

Guidelines  for  Worksheets  CM2  and  Cl. 13 

If  you  plan  to  collect  instructor  reactions,  use  Worksheet  CL  12  or 
CL 1 3,  the  Instructor  Reaction  Questionnaire,  and  interview  forms. 

Step  CL3  generally  applies  to  their  use.  There  are  two  major 
differences: 
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•  The  questions  the  instructor  is  able  to  answer  relate  to  the 
accuracy  of  the  content  and  the  adequacy  of  the  materials  in 
terms  of  their  ability  to  help  him  teach  the  course. 

•  You  may  prefer  to  obtain  all  instructor  reactions  via  interviews, 
particularly  if  there  are  only  one  or  two  instructors  involved  in 
the  tasks  you  selected  to  analyze. 

If  you  plan  to  ask  Master  List  questions  of  instructors  in  hard-to- 
observe  situations,  follow  the  procedure  in  Step  C1.6,  but  include  the 
title  “INSTRUCTOR  INTERVIEW  FORM”  prior  to  reproducing  the  form. 
Again,  only  the  TEE  analyst  or  a  well-trained  associate  analyst  should 
conduct  the  interviews. 

Note  that  instructors  are  usually  subject  matter  experts,  and  that  a 
number  of  TEE  questions  (which  apply  in  both  observable  and  hard-to- 
observe  situations)  may  require  a  subject  matter  expert’s  help.  You 
may  wish  to  include  some  questions  of  that  nature  on  instructor 
reaction  instruments,  especially  if  you  have  no  associate  analyst.  Those 
worksheets  would  then  be  used  with  every  applicable  training  event,  and 
the  Master  List  questions  on  them  would  be  left  off  of  the  regular 
data  collection  worksheets  (Cl.l  to  C1.8). 

C1.8  If  circumstances  warrant,  tailor  your  own  process  evaluation  worksheets 

to  the  training  settings  you  will  observe.  As  an  example,  if  only  one 
class  was  in  residence  for  the  course  you  were  to  evaluate,  you  migl.v 
decide  to  evaluate  the  entire  course  yourself  or  with  the  assistance  of 
an  associate  analyst  (defined  in  the  introduction).  Several  questions 
have  been  left  out  of  the  process  data  collection  worksheets  simply 
because  they  require  data  collectors  to  classify  content  type,  a  skill 
deemed  difficult  to  train  for  most  personnel.  You  would  want  to 
include  some  of  these  questions  on  your  worksheets  in  the  situation 
described.  (They  are  marked  with  an  ANALYST  ONLY  designation 
found  in  the  upper  right-hand  corner  of  many  Master  List  questions  in 
Appendix  A.) 

Task  C2:  Make  logistical  arrangements  to  conduct  the  training  process  TEE. 

C2.1  Find  Worksheet  C2  in  Appendix  H,  you-  course  outline  showing  training 

events  selected  for  observation  in  Task  A5,  and  a  course  schedule. 

List  the  training  events  you  selected  for  observtion  on  Worksheet  C2. 

You  may  not  need  to  observe  every  event  selected,  although  that  is  the 
best  course  of  action  to  insure  the  validity  of  the  TEE.  You  must 
observe  at  least  the  following  events  for  those  tasks  selected  for 
evaluation: 

•  A  “Final  Test”  for  each  task  selected  for  evaluation.  If  it 
involves  both  written  and  practical  tests,  both  should  be 
observed. 

•  At  least  50%  of  the  demonstrations  for  each  instructor  involved. 
A  large  number  of  small  group  demonstrations  may  be  listed  as 
one  event  on  your  worksheet.  The  overall  event  must  be 
observed  for  each  task  where  it  occurs. 
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•  As  much  practice  as  is  practical  for  each  task  selected.  Each 
instructor  giving  feedback  and  each  different  piece  of  equipment 
should  be  observed  an  equal  amount  of  time.  Practice  must  be 
observed  for  each  task  when  it  occurs. 

•  As  much  classroom  instruction  as  possible;  an  equai  number  of 
events  for  each  instructor. 

•  As  many  of  the  other  training  event  types  as  possible  or  those 
that  are  necessary  to  fulfill  any  specific  directions  for  the  TEE 
desired  by  your  sponsor. 

When  your  purpose  is  to  certify  training  prior  to  an  operational  test 
(OT)  of  a  developing  weapons  system,  it  is  more  important  to  observe  a 
large  percentage  or  ail  of  the  training  events  for  their  entire  duration. 
On  the  other  hand,  when  you  are  observing  ongoing  training  that 
appears  tn  be  functioning  fairly  well  and  which  is  consistent  across 
instructors,  complete  observation  of  every  training  event  becomes  less 
important. 

From  the  course  schedule  or  in  an  interview  with  an  instructor  or 
course  manager,  record  the  date,  class  #,  time,  location,  objective  //, 
and  contact  for  each  event.  (See  the  guidelines  at  the  end  of  this 
task.)  It  may  be  necessary  in  some  TEEs  to  do  so  as  the  TEE  is 
conducted.  Also  make  a  list  of  the  instructors  and  assistant  instructors 
and  assign  each  one  a  code  number. 

Communicate  your  plans  for  an  on-site  visit  to  the  personnel  in  charge 
of  the  training  to  be  observed.  Ask  that  arrangements  be  made  for 
you  to  have  access  to  the  training  and  testing  events  you  selected  to 
observe.  It  is  a  good  idea  to  relate  the  following  to  the  training 
deliverers: 

•  Your  primary  role  is  to  evaluate  the  training  design  and  methods 
-  it  is  not  to  evaluate  the  trainees  or  the  instructors. 

•  You  selected  the  training  and  testing  events  that  are  the  most 
appropriate  to  your  TEE.  Not  all  events  will  necesarily  be 
observed,  but  the  ones  you  selected  are  "musts." 

•  You  will  not  interfere  with  the  conduct  of  the  training  and  the 
testing. 

•  You  will  insure  that  the  trainees  and  instructors  remain 
anonymous,  if  at  all  possible. 

•  You  will  need  to  be  able  to  interview  the  instructors  from  time 
to  time  and  at  least  a  sampling  of  the  trainees  (if  these 
activities  are  a  part  of  your  TEE). 

•  Who  you  will  give  your  findings  to  and  what  they  will  b*  used 
for. 


C2.4  Review  Phase  A  Worksheet  A1  information.  Make  sure  that  you  have 

the  "Resource  Material"  listed.  If  you  do  not  have  the  training 
material  to  be  used  in  the  actual  training  and  the  test  instruments, 
contact  the  "Source"  listed  on  the  worksheet  to  obtain  them. 

C2.5  Review  the  training  and  testing  materials  to  familiarize  yourself  with 

the  content  of  the  training.  Concentrate  on  those  materials  that  relate 
to  the  performance  tasks  and  training  events  that  you  have  planned  to 
evaluate  in  detail  (selected  in  task  A5). 


Guidelines  for  Worksheet  C2 


Here  are  some  guidelines  for  using  Worksheet  C2. 


ITEM 


GUIDELINES 


COURSE 

DATE/TIME 

CLASS  // 

TYPE  OF  EVENT 

OBJECTIVE  // 


The  name  of  the  course  for  your  TEE 

Specific  time  of  the  training  or  testing  event  you'll 
observe 

The  number  of  the  class  or  company  to  be  observed, 
e.g.,  S2-3  or  A-3-1 

Event  types  specified  on  your  course  outline  from 
Task  A3 

The  TLO  or  LO  numbers  on  your  objectives  list  or 
course  outline 


CONTENT 

LOCATION 

CONTACT 

MATERIALS 

NOTES 


The  lesson  title  from  your  course  outline.  Include 
the  lesson  or  task  number  if  you  so  desire. 

Specific  place  where  the  event  will  occur 
(Installation,  Building,  Target  Range,  etc.) 

Name  of  instructor  or  person  in  charge  of  the  event 

List  of  trainee  materials,  test  instruments,  etc. 
which  will  be  used  in  the  training  event 

Space  for  you  to  make  any  reminders  to  yourself 
such  as: 

—  need  for  obtaining  materials  in  advance 

—  need  to  set  up  observation  with  the  contact, 
etc. 
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PHASE  D:  CONDUCT  TRAINING  PROCESS  EVALUATION 
Overview  of  Phase  D 

You  will  collect  three  major  types  of  information  during  this  phase: 

•  Direct  observation  data.  This  is  information  about  the  training 
environment,  use  of  materials,  and  trainee/instructor  activites  which 
can  be  collected  during  the  training  and  testing.  You  will  train 
data  collectors  to  use  the  worksheet  you  prepared  in  Task  Cl  to 
obtain  this  data,  unless  you  intend  to  collect  it  yourself  or  with 
your  assoc  fate  analyst. 

•  Trainee  and  instructor  characteristics.  This  is  information  about 
trainees  and/or  instructors  that  may  influence  training  results. 
Training  data  should  always  be  obtained  if  possible;  instructor  data 
are  optional  -  particularly  if  the  course  is  taught  by  only  one  or 
two  people.  It  is  recommended  that  general,  or  group,  trainee  data 
be  collected.  For  example,  rather  than  record  and  document  the 
educational  level  of  each  trainee,  you  might  quickly  review  records 
and  record  general  information,  such  as  ’’trainees  range  from  Sth 
grade  to  2nd-year  college;  almost  all  are  high  school  graduates." 
If  you  find  that  some  trainees  are  doing  very  poorly  and  others 
very  well  -  and  you  suspect  lack  of  education  as  a  reason  for 
difficulty  -  you  might  get  more  detailed  educational  data  to  make 
a  comparison  and  check  your  hypothesis.  If  the  course  involves 
crew  training,  you'll  need  to  describe  the  characteristics  of  the 
trainees  for  each  job  the  crew  performs. 

•  Trainee  and  instructor  reaction  data.  This  is  information*  gathered 
at  the  end  of  training  and  testing  events,  about  trainee  and  instructor 
opinions  of  the  training*  Sample  reacton  instruments  are  provided 
in  the  guidelines  which  you  will  have  selected  in  Task  Cl  if  this 
type  of  data  is  to  be  collected  in  your  TEE. 

Here  are  the  tasks  of  Phase  D: 


TASKS: 

Dl.  Train  data  collectors 


D2.  Collect  data  on  training 
and  testing  events 

D3.  Collect  trainee  and 

instructor  characteristics 
data 


RELEVANT  WORKSHEET: 

Training  materials  in  the  Data 
Collector's  Manual  and  Appendix  F 

C2  filled  out  previously,  work¬ 
sheets  created  in  Task  Cl 

D3.1 

D3.2 
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Guidelines  lor  Phase  D 


Task  Dl:  Assemble  and  Train  Data  Collectors. 

Di.l  It  is  assumed  that  a  number  of  data  collectors  can  be  made  available 

to  you  during  this  phase  if  needed.  (They  will  be  needed  only  if  the 
number  of  training  events  to  be  observed  at  any  given  time  demands 
it.)  If  you  decide  to  use  them,  you  should  request  them  through  your 
sponsor  far  enough  in  advance  for  them  to  be  available.  If  you  can 
obtain  one  subject  matter  expert  (SME)  for  the  course  you  are 
evaluating,  it  will  be  to  your  advantage.  In  general,  however,  your 
data  collectors  should  not  be  SMEs,  but  should  be  familiar  with 
standard  Army  Training  Procedures.  It  will  also  be  very  helpful  if  your 
data  collectors  can  reacf  well  enough  to  understand  the  questions  in  the 
Master  List  without  much  difficulty.  The  number  of  data  collectors 
needed  depends  on  the  logistics  of  the  course  and  the  nature  of  your 
TEE.  Consider  the  following  points: 

•  The  number  of  training  events  selected  for  observation  in 
Task  A  5 

•  Whether  more  than  one  class  is  in  training  for  the  course  and 
how  their  schedules  overlap 

•  Your  deadline  for  completing  the  TEE 

•  The  fact  that  some  data  collectors  will  not  collect  adequate  data 

D1.2  In  order  to  proceed  at  this  point  you  must  have  completed  all  tasks  in 

Phases  A  and  C.  Gather  the  appropriate  documentation  and  obtain 
sufficient  copres  of  the  items  needed  for  your  data  collectors  to 
observe  each  training  event.  You  should  have: 

•  Your  course  outline  from  Task  A5  showing  the  training  events 
selected  for  observation 

•  Enough  copies  of  the  worksheets  constructed  in  Task  Cl  to 
conduct  the  process  TEE 

•  Worksheet  C2  and  enough  copies  of  the  training  materials  listed 
on  it  for  the  data  collectors  to  use  during  observations 

»  The  list  of  instructors  and  their  code  numbers  from  Task  C2 

•  A  Data  Collectors  Manual  for  each  data  collector 

D1.3  Begin  the  training  by  explaining: 

»  What  a  TEE  is 

•  What  will  happen  to  the  results  and  how  they  might  be  used 

•  The  importance  of  the  data  collectors  performing  their  tasks 
correctly 
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Explain  what  the  data  collectors  will  be  required  to  do.  Discuss: 

•  Observation  of  training  events,  in  which  ratings  and  observations 
are  made 

•  General  daily  procedures  (outlined  in  Step  D2.1) 

•  The  mechanics  of  how  to  use  the  worksheets  and  how  they  are 
tied  to  the  Data  Collector's  Manual.  Introduce  each  worksheet 
you  plan  to  use. 

Compile  a  list  of  data  collectors  if  one  is  not  available  to  you,  and 
pass  out  the  Data  Collector's  Manual,  one  to  each  data  collector. 

Pass  out  copies  of  the  names  and  code  numbers  of  the  instructors. 

Show  the  data  collectors  where  their  names  and  the  instructors' 
numbers  go  on  the  worksheets. 

Conduct  training  on  technical  terminology  used  in  the  data  collection 
worksheets.  The  evaluation  questions  to  be  used  in  the  TEE  contain 
some  of  the  terms  found  in  the  Glossary  of  this  User's  Guide.  The 
appliable  Glossary  terms  are  found  in  the  Data  Collector's  Manual  (in 
logical  rather  than  alphabetical  order).  Explain  each  of  these  terms 
and  discuss  them  with  the  data  collectors.  Allow  time  for  them  to 
study  or  discuss  this  information,  either  individually  or  in  groups.  After 
that,  call  on  the  data  collectors  to  explain  the  terms  in  their  own 
words.  Review  the  terms  further  if  necessary.  Finally,  it  is  suggested 
that  you  give  them  a  written  quiz  to  insure  mastery  of  the  terms. 

Unless  the  questions  on  your  worksheets  deal  with  only  one  task  level, 
conduct  the  training  on  how  to  classify  task  levels  (see  the  Data 
Collector's  Manual).  Instructon  on  how  to  classify  content  type  is  also 
included  in  your  version  of  this  instruction  in  Appendix  F  ,  but  the 
data  collectors  will  not  normally  need  to  learn  to  classify  content 
types.  It  is  therefore  not  included  in  their  manual.  Allow  time  lor 
the  data  collectors  to  read  the  instruction  on  task  level  and  do  the 
practice  items  with  feedback.  Then,  go  over  each  of  the  Objectives 
for  Discussion  on  Task  Level.  Ask  the  data  collectors  to  classify  each 
one  and  tell  you  why  they  classified  them  as  they  did.  The  feedback 
to  your  self  test  in  Appendix  F  corresponds  to  these  objectives. 

Finally,  ask  the  data  collectors  to  write  the  task  level  of  each  of  the 
objectives  titled  Final  Practice  on  Task  Level.  Discuss  each  answer. 
The  feedback  to  your  additional  practice  objectives  in  Appendix  F 
corresponds  to  those  objectives. 

If  your  data  collectors  arc  not  subject  matter  experts  (SMEs),  have 
your  associate  analyst  or  another  5ME  give  a  short  series  of  lectures 
on  the  tasks  to  be  evaluated.  He  should  not  go  into  great  detail,  but 
should  help  everyone  to  become  familiar  with  the  task  actions, 
conditions,  and  standards.  This  knowledge  will  be  necessary  to  properly 
fill  out  the  data  collection  worksheets. 

Go  over  each  question  included  on  your  worksheets  by  explaining  the 
question  and  giving  an  example  or  two  of  how  to  make  a  rating  from 
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the  course  under  evaluation.  Examples  can  be  found  in  the  Master  List 
of  Evaluation  Questions.  Give  examples  for  as  many  questions  as  you 
can  from  the  Master  List  and  your  own  familiarity  with  the  course. 
Your  associate  analyst  should  be  able  to  help  in  supplying  examples. 
After  each  TEE  question  is  covered,  ask  questions  of  your  data 
collectors  and  answer  any  questions  they  ask. 

D1.9  Demonstrate  how  to  give  trainee  interviews  and  how  to  administer 

questionnaires,  if  they  are  a  part  of  your  TEE.  Help  data  collectors  to 
understand  the  following  considerations: 

•  Trainee  reaction  instruments  which  include  Master  List  questions 
should  be  administered  immediately  after  the  events  to  which 
they  apply.  Otherwise  the  trainees  will  be  likely  to  forget  what 
actually  happened  or  the  memory  of  other  events  will  interfere. 
The  standard  reaction  instruments  (Worksheets  Cl.  10  and  Cl.  11) 
however,  should  be  administered  only  once  or  twice,  near  the 
middle  and/or  at  the  end  of  the  course. 

•  You  should  get  the  standard  reactions  from  only  a  sample  of 
trainees  for  a  sample  of  training  events.  The  same  students 
should  not  be  used  more  than  once  or  twice  (they  could  be  used 
a  second  time  a  few  weeks  later). 

•  When  administering  questionnaires,  trainees  should  be  required  to 
stay  until  everyone  is  finished.  Otherwise,  some  trainees  will 
tend  to  rush  through  the  questionnaire  and  give  less  complete 
answers. 

•  When  using  questionnaires,  the  trainees1  anonymity  should  be 
insured,  if  possible.  Reactions  are  very  easy  to  "fake"  and 
trainees  often  want  to  please  evaluators  by  expressing  what  they 
think  others  what  to  hear.  This  tendency  can  be  reduced 
somewhat  by  having  trainees  express  their  reactions  anonymously. 

•  Be  sure  that  data  collectors  will  label  each  batch  of 
questionnaires  with  the  information  called  for  in  the  headings  to 
your  other  worksheets. 

•  Wlien  administering  questionnaires  which  include  Master  List 
questions  you  should  also  be  sure  the  trainees  understand  the 
extent  of  the  training  event  they  are  being  questioned  about. 

That  is,  they  should  answer  only  for  the  one  training  event  (or 
events  if  you  desire)  which  have  just  taken  place. 

NOTE:  (1)  From  test  results  and  class  participation  you  should  have  begin  to  form 
an  idea  of  who  your  good  data  collectors  are  and  those  wiio  will  not 
collect  adequate  data.  You  should  make  a  few  confidential  notes  on 
your  data  coilec:ors  list. 

NOTE:  (2)  Normally  only  the  TEE  analyst  or  associate  analyst  would  administer 
reaction  instruments  to  instructors. 


D1.10  Give  your  data  collectors  practice  in  making  ratings  using  your 

worksheets,  if  at  all  possible.  Arrange  to  observe  training  events  not 
selected  for  your  TEE  or  from  another  course.  Go  with  small  groups 
of  data  collectors  and  have  them  fill  out  worksheets  that  apply  to  the 
event.  Have  every  data  collector  observe  each  type  of  event  to  be 
evaluated  in  your  TEE  at  least  once  if  possible.  Make  your  own 
ratings  for  each  event  as  well. 

Collect  the  worksheets  after  each  event  and  compare  them  to  your  own 
ratings.  Make  notes  on  patterns  you  see  which  diverge  from  your 
ratings  and  attempt  to  explain  them  in  terms  of  the  following: 

•  rating  what  is  taught  rather  than  how  it  is  taught 
o  misunderstanding  of  questions 

•  a  good  or  bad  impression  of  the  students  rather  than  their 
performance 

•  the  data  collectors  being  too  lenient  or  too  hard 

After  each  type  of  event  has  been  rated  in  practice  and  you  have 
evaluated  the  ratings,  meet  with  each  group  if  numbers  permit,  or  with 
the  entire  group  of  data  collectors.  Discuss  your  evaluation  of  their 
ratings  with  them. 

You  may  wish  to  repeat  this  step  one  or  more  times,  depending  on  how 
close  the  data  collectors’  ratings  were  to  your  own. 

Dl.ll  The  following  "training  plan"  for  a  data  collector's  workshop  is  provided 

as  a  guide.  The  workshop  should  be  scheduled  for  three  consecutive 
days,  i.e.,  Monday -Wednesday,  Tuesday -Thursday,  or  Wcdnesday-Friday. 
Try  not  to  break  the  workshop  up  over  a  weekend.  Schedule  for  three 
full  days.  You  may  not  use  all  of  the  third  day  but  the  first  time  you 
fail  to  schedule  all  of  it  will  be  the  one  time  you  will  need  it. 

Arrange  the  workshop  in  a  fashion  similar  to  the  following: 

•  Day  !.  Morning.  Introduction  and  Overview  of  TEE  System 

-  Introduce  personnel. 

-  State  the  purpose  of  the  workshop. 

-  Collect  background  information  on  all 
attendees. 

-  Distribute  materials  (c.g.,  Data  Collector’s 
manual,  worksheets,  etc.). 

-  Provide  overview  of  TEE  System  (based  on 
discussion  in  Section  I  of  TEE 
Evaluator's  Handbook). 

•  Day  1.  Afternoon.  Terminology  and  Familiarisation 

-  Supervise  self-instruction  on  terminology 
(using  material  from  Data  Collector’s 
Manual). 
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-  Discuss  terminology  and  conduct  oral  and 

written  quizzes.  Provide  feedback  and 
remediation. 

-  Supervise  self-instruction  on  classification 

of  task  levels  (using  materials  from 
Data  Collector’s  Manual). 

-  Discuss  classification  of  task  levels  and 

conduct  oral  and  written  quizzes. 
Provide  feedback  and  remediaton. 

-  Provide  overview  of  worksheets  and 

discuss  Master  List  Questions. 


•  Day  2.  Morning.  Guidance  on  Ratings  and  Recording  Data 

-  Review  each  Master  List  Question  and 

provide  detailed  rating  guidance. 

-  Discuss  the  "mechanics”  of  recording 

data  on  each  worksheet. 

-  Answer  questions  on  all  materials  and 

procedures  covered  up  to  this  point  in 
workshop. 

-  Introduce  procedures  for  use  of 

questionnaires. 

•  Day  2.  Afternoon.  Supervised  Training  Observation  and  Data 

Collection 


-  Prepare  data  collector  for  exercise  in 

observing  training,  recording  data,  and 
administering  questionnaires. 

-  Supervise  sample  data  collection. 

-  Evaluate  worksheet  entries  recorded  by 

each  data  collector. 

-  Discuss  observations  and  data  entries  from 

exercise. 

-  Provide  remediation  as  necessary. 


•  Day  3.  Morning.  Data  Collection  Practice 

-  Organize  data  collector  into  groups  and 

send  out  for  training  observation  and 
data  collection.  Allow  for  coverage  of 
each  worksheet  if  possible.  Also 
Include  the  administration  of 
questionnaires. 

-  Review  each  data  collector’s  responses. 

•  Day  3.  Afternoon.  Final  Review  and  Selection 

-  Discuss  each  data  collector’s  observations 

and  data  entries  individually. 

-  Summarize  the  workshop  experience  for 

the  group. 

-  Select  data  collectors  for  TEE. 
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Task  D2:  Collect  data  on  training  and  testing  events. 

D2.1  Prior  to  each  day  of  observation,  make  any  necessary  final  arrange¬ 

ments  with  your  points  of  contact  and  prepare  to  issue  the  next  day's 
assignments  to  your  data  collectors  or  to  make  observations  yourself. 
Arrange  the  necessary  course  materials  which  you  or  your  data  collec¬ 
tors  will  need  (os  noted  on  Worksheet  C2)  and  supply  sufficient  copies 
of  the  appropriate  data  collection  worksheets  for  each  training  event  to 
be  observed.  You  may  wish  to  prepare  written  assignments  for  data 
collectors  showing  the  date,  time,  class  //,  type  of  event,  objective  //s, 
content,  location,  and  contact  for  the  events.  You  could  photocopy 
Worksheet  C2  and  cut  it  into  strips  containing  the  data  for  each  event. 
Write  the  data  collectors'  names  on  the  assignments  if  you  do  this,  but 
in  any  event  be  sure  to  keep  your  own  record  of  the  data  collector  (s) 
assigned  to  each  event  in  the  Notes  column  of  Worksheet  C2. 

As  discussed  in  Step  C2.1,  decide  how  many  segments  of  different  dem¬ 
onstration  and  practice  events  on  the  same  tasks  can  practically  be 
observed  and  include  such  instructions  in  your  assignments  to  data 
collectors.  Also,  it  is  best  for  data  collectors  to  stay  for  an  entire 
training  event,  although  that  is  absolutely  necessary  only  in  the  case  of 
a  few  questions  related  to  practice  events,  e.g.,  "Do  all  students  prac¬ 
tice?"  Also,  if  applicable,  be  sure  that  the  conventions  for  trainees 
and  instructor  reaction  instruments  covered  in  Step  D1.9  are  followed. 

At  first,  assign  data  collectors  in  groups  of  2  or  3.  If  possible  the 
group  should  contain  at  least  one  "reader"  and  one  subject  matter 
expert.  At  the  end  of  each  day  you  can  compare  ratings  within  groups 
in  an  attempt  to  identify  your  best  data  collectors,  those  who  do  an 
adequate  job,  and  those  who  do  not  collect  good  data.  You  should  use 
these  judgments  in  making  future  assignments.  Your  best  data  collectors 
might  be  trusted  to  cover  events  alone  (where  the  amount  of  observa¬ 
tion  permits).  Poor  data  collectors,  however,  should  not  be  assigned  to 
events  alone  unless  the  events  would  otherwise  be  missed  altogether. 

It  could  be  unwise,  however,  to  drop  data  collectors  from  the  THE 
unless  that  would  dearly  be  a  less  desirable  fate  than  continuing,  since 
it  may  adversely  affect  the  performance  of  other  data  collectors. 

D2.2  Give  the  assignments  to  your  data  collectors  with  all  necessary  explana¬ 

tions  and  have  them  report  back  to  you  when  they  finish.  Upon  their 
return,  collect  their  worksheets  and  insure  that  all  heading  information 
is  correct  and  that  worksheets  arc  filled  out  completely.  Question  your 
data  collectors  about  problems  with  these  considerations  and  have  them 
correct  them  and  any  other  obvious  flaws.  Discuss  their  "2"  and  "3" 
ratings  and  insure  that  their  comments  arc  sufficiently  complete.  You 
will  probably  find  that  some  data  collectors  are  able  to  verbally  report 
events,  but  may  not  be  able  to  write  them  down.  These  persons  may 
still  be  usable  provided  that  you  are  able  to  work  closely  with  them. 

D2.3  As  the  worksheets  are  turned  in  or  after  they  have  been  collected  each 

day,  examine  each  worksheet  and  decide  whether  the  data  are  usable  or 
not.  If  data  are  not  usable,  you  may  be  able  to  assign  other  data 
collectors  to  observe  the  event  it  it  has  not  already  ended,  such  as 
events  which  are  carried  over  to  the  next  day.  Do  thin  only  if  ii  is 
needed  tu  cover  events  required  in  Task  C2. 
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File  unusable  data  separately. 


D2.4  You  have  probably  decided  to  use  Worksheet  Cl. 9,  which  is  used  to 

rate  questions  which  involve  a  series  of  training  events.  If  so, 
whenever  such  a  series  is  completed  for  a  task  (e.g.,  events  containing 
a  statement,  examples,  and  practice),  sit  down  with  the  applicable  data 
collection  worksheets  and  fill  out  Worksheet  Cl. 9.  You  may  wish  to 
have  the  associate  analyst  or  a  data  collector  do  this,  in  which  case  it 
is  best  to  assign  the  same  person  to  do  so  for  the  entire  TEE. 

Task  D3:  Collect  trainee  and  instructor  characteristics  data. 

Guidelines  for  Worksheet  D3.1 

D3.1  Make  as  many  copies  of  Worksheet  D3. 1  as  you  have  DUTY  POSITIONS 

in  the  course. 

D3.2  Check  the  TYPE  of  course:  Initial,  refresher,  or  transition  training. 

D3.3  Record  DUTY  POSITION  and  MOS  and  the  NUMBER  of  trainees  in 

each. 

D3.'i  Use  documentation  and  o-  her  sources/records  to  characterize  the 

trainees  in  the  course.  .ote  on  Worksheet  D3.1  that  there  is  a 
"Spt  .  Tied"  column  and  an  "Actual"  column.  Any  discrepancies  you  find 
between  the  specified  characteristics  of  the  trainee  and  those  that 
actually  attended  may  become  important  later  when  you  artf  trying  to 
determine  reasons  for  performance  deficiencies. 


CATEGORY 

GUIDELINES 

Grade 

(Self-explanatory) 

Academic 

Highest  educational  level  achieved 

Age 

(Self-explanatory) 

Years  In  Army 

(Self-explanatory) 

Previous  job 
experience 

What  other  duty  positions  have  the  trainees 
held  or  should  have  held? 

Special  aptitudes/ 
qualifications 

Are  there  any  special  requirements  specified  for 
entry  into  the  course? 

Method  of  selection 
for  this  training 

For  example,  did  the  trainee  need  to  pass  an  entry 
test?  Be  qualified  in  some  other  Duty  Position? 

Course  grades  from 
similar  MOS 

If  the  course  is  transition  training  from  an  older 
MOS,  how  did  the  trainee  perforin  in  his  previous 
training? 

Commander's  per¬ 
formance  report  $1 
recommendations 

If  motivation,  flexibility,  or  other  characteristics 
are  important,  these  can  be  a  valuable  indicator  of 
trainee  qualifications. 

CATEGORY 


GUIDELINES 


Other  Your  documentation  may  list  other  requirements 

for  the  trainee  over  and  above  those  listed  on  the 
worksheet.  Or,  you  may  wish  to  note  other  facts 
about  the  trainee  which  bear  on  the  TEE. 


Guidelines  for  Worksheet  D3.2 

D3.5  Make  as  many  copies  of  Worksheet  D3.2  as  there  are  instructors  for 

the  course. 

D3.6  Record  the  instructor  code  numbers  on  each  copy  of  the  worksheet. 

D3.7  Record  the  number  of  times  the  instructor  has  given  the  course. 

D3.S  Use  the  guidelines  below  to  record  “Specif ied,:  and  "Actual” 

characteristics  of  the  instructors. 


CATEGORY 

GUIDELINES 

Grade 

(Self-explanatory) 

Age 

(Self-explanatory) 

Years  in  Army 

(Self-explanatory) 

Training  in 
training  delivery 

What  formal  instructor  training  should/did 
the  instructor  undergo? 

Experience  in 
training  delivery 

How  long  has  the  instructor  been  an  instructor? 
Note  any  other  applicable  training  experience  either 
specified  and/or  actual. 

Experience  in 
subject  matter 

Has/should  the  instructor  have  had  field  experience 
in  the  tasks  of  the  course?  To  what  degree  is/should 
the  instructor  be  an  expert  in  the  tasks  of  the 
course? 

Involvement  with 
course  development 

Did  the  instructor  personally  develop  the  course 
or  work  with  course  developers  in  the  design  of 
the  course? 

Involvement  with 

hardware 

development 

Was  tin*  instructor  involved  in  designing  or 
developing  parts  of  the  weapon  system? 

Special  aptitudes/ 
Qualifications 

Arc  any  unusual  abilities,  skills,  or  experiences 
required? 
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PHASE  E:  ASSESS  TRAINEE  PERFORMANCE 


Overview  of  Phase  E 

This  phase  has  four  major  purposes: 

1.  Assess  how  well  trainees  can  perform  the  tasks  selected  for 
evaluation  upon  completion  of  training. 

2.  Judge  the  adequacy  of  the  above  assessments. 

3.  Summarize  observations  of  the  training  and  training  materials  as 
an  aid  to  identifying  performance  discrepancies  and  as  input  to 
the  revision  process. 

4.  Identify  tasks  on  which  performance  standards  are  not  met 
("performance  discrepancies"). 

You  should  remember  that  assessment  of  the  quality  of  trainee 
performance  will  almost  always  be  an  estimate.  Accurate  assessment 
of  the  quality  of  performance  requires  direct  observation  of  the 
performance  in  the  real-world  situation.  In  most  TEEs,  this  is 
impossible.  Trainee  performance  will  be  assessed  under  circumstances 
which  approximate,  with  varying  degrees  of  realism,  the  actual 
conditions  under  which  the  trainees  are  ultimately  expected  to  perform. 

It  should  be  made  clear  that  measures  of  student  performance  provide 
the  most  accurate  and  meaningful  assessment  of  the  adequacy  of 
training.  Other  TEE  observations  may  be  used  to  supplement  decisions 
based  on  performance  Of  performance  is  marginal  or  the  measure  is  not 
completely  valid).  However,  such  observations  should  usually  be 
considered  as  a  secondary  source  of  information  for  evaluation  purposes. 
On  the  other  hand,  those  observations  are  the  primary  source  in 
identifying  the  required  revisions  in  training,  once  the  discrepant  task 
performances  are  identified.  The  primary  output  of  TEE  Phase  E  is  a 
list  of  tasks  for  which  the  standards  have  not  been  met  — 

"performance  discrepancies"  —  and  a  list  of  potential  problem  areas  for 
each  task  —  "training  deficiencies."  Each  deficiency  will  have  been 
rated  as  minor  or  serious  as  it  would  impact  test  adequacy  or  student 
performance. 

Here  are  the  TEE  tasks  of  Phase  E: 


TASKS* 


RELEVANT  WORKSHEET: 


Ei.  Collect  and  summarize  test  data.  El  (Appendix  H) 


E2.  Summarize  product  and  process 
evaluation  data. 


E3.  Identify  task  and  team  function 
performance  discrepancies. 


E2.1  (Appendix  H) 
E2.2  (Appendix  H) 
E2.3  (Appendix  H) 

E3.1  (Appendix  H) 
E3.2  (Appendix  H) 
E3.3  (Appendix  H) 
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Guidelines  for  Phase  E 


Task  El:  Collect  and  summarize  test  data. 

The  most  important  test  data  treated  in  this  task  are  from  the 
trainees*  first  attempt  at  the  final  test  for  the  course  or  each  course 
unit.  If  there  is  not  final  test  or  series  of  final  unit  tests,  you  should 
try  to  obtain  scores  from  performance  tests  on  lessons,  if  they  were 
given,  and  combine  these  into  a  final  test  for  the  purposes  of  the  TEE. 
In  any  case,  the  use  of  one  composite  set  of  data  per  trainee  is 
advised. 

We  make  these  assumptions  regarding  test  data: 

•  The  trainees  performance/outputs  have  been  "scored,"  and  you 
have  access  to  the  raw  data. 

•  Their  performance  in  some  sort  of  operational  event  was 
recorded,  as  opposed  to  a  written  examination  (unless,  of  course, 
written  performance  was  the  only  ’'level  of  reality"  assessed). 

•  Personnel  other  than  yourself  collected  the  test  data. 

•  Most  of  the  scoring  was  of  the  "Pass/Fail"  or  "Go/No  Go" 

variety,  or  the  scoring  that  was  done  can  be  easily,  converted  to 
such.  r 

•  T^e  scorers  made  their  assessments  based  on  the  same 
performance  or  output  standards. 

•  Scores  from  at  least  10  trainees  are  available. 

El.l  Obtain  and  assemble  the  raw  data  produced  in  the  final  test.  We 

assume  that  it  will  be  in  the  form  of  "Go/No  Go"  or  "Pass/Fail" 
scoring.  If  not,  convert  the  scores  to  that  type  of  scoring.  If  the  test 
Is  a  written  test  or  has  separate  items  for  the  same  task  or  TLO,  you 
will  have  to  identify  the  items  for  each  task  and  the  correpsonding 
pass/fail  criteria  for  each  task  (when  the  test  is  scored  only  as  a 
whole)  In  order  to  convert  to  Go/No  Go  scoring. 

E1.2  Worksheet  El  presents  one  format  for  summarizing  the  data  for  later 

analysis.  Make  as  many  copies  of  Worksheet  El  as  you  need  to 
summarize  the  test  performance  for  each  task  and/or  team  function 
being  evaluated. 

Use  the  guidelines  in  the  table  following  Step  El. 3  below  in  filling  out 
Worksheet  El.  If  both  Individuals  and  crews  were  scored,  use 
worksheets  for  individual  tasks  separate  from  those  for  team  functions. 

E1.3  Repeat  Steps  El.l  and  El.2  for  any  other  types  of  tests  you  are 

evaluating. 
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Guidelines  for  Worksheet  El 


WORKSHEET  ITEM 


GUIDELINES/NOTES 


Course  Course  title  whose  selected  tasks  you  are  evaluating. 

Test  Period  The  dates  of  the  test. 


Test  Conditions/  Describe  the  conditions  in  which  the  test  was  conducted. 

Situation  Was  it  in  the  classroom  situation?  Full-scale  operational 

trial? 


Type  of  Test 
Code  Number 


Positions 


Individual  or 
Crew  Numbers 


Tasks  Selected 
for  Evaluation 


Number  f  NO  GO” 


Number  of  Crews/ 
Individuals  Tested 


Check  appropriate  box. 

Give  the  code  numbers  of  the  instructors  and/or  scorers 
who  were  involved  in  the  test  of  the  tasks  you  are 
evaluating. 

Hob  title(s)  of  the  trainees  or  members  of  the  crews 
involved  in  the  test.  If  different  positions  have 
different  tasks,  group  the  tasks  for  each  position  or  use 
separate  sheets  for  each. 

* 

Assign  a  code  number  to  each  trainee  who  was 
scored,  or  a  number  or  letter  to  the  crews. 

List  all  trainees  or  crews  that  were  scored  in  the 
first  column  of  the  worksheet. 

Record  the  number  for  each  task  or  team  function 
selected  in  Task  A3  (one  per  column  across  the  top 
of  the  chart).  Copy  as  many  as  necessary.  Enter 
the  word  "TEST"  after  the  individual  tasks  and/or 
after  the  team  functions. 

For  each  individual  or  crew,  record  the  result  of  the 
test  for  the  first  trial  of  each  task  scored:  A  check 
for  "Go"  or  "Pass.”  An  X  for  "No  Go  or  "Fail." 

Count  the  number  of  "No  Go"  ("X")  for  each  task  or 
team  function.  If  there  is  evidence  that  a  trainee 
received  a  "Go"  only  after  hints  or  coaching,  convert 
that  score  to  a  "No  Go." 

The  total  number  of  individuals  or  crews  tested  in 
the  tasks  you  are  analyzing,  (Enter  one  figure  and 
circle  it  if  fits  alt  taski.) 


%  NO  GO 


Divide  the  number  NO  GO  by  the  number  tested  for 
each  task  or  team  function. 


"TEST" 


In  a  column  labelled  "test,"  summarize  individual  and 
crew  scores  for  the  test  as  whole. 


NOTE:  See  the  examples  of  Worksheet  El  on  the  next  two  pag^s. 
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Type  of  Test: 


Individual  caf 
Final  ET" 
Written  □ 


Position:  J) 


re i/e  r 


Crew  □ 

Within-Course  □ 
Hands-On 

•-  Co  *rt  m &  ncl* < 


Code  No:  ^,i  7 
Entry  Test  □ 
Pretest  □ 


Task  E2j  Summarize  Product  and  Process  Evaluation  Data. 


E2.1 


Locate  any  trainee  reaction  instruments  (Worksheet  Cl.  10  or  Cl.  11) 
filled  out  in  the  process  evaluation.  For  each  trainee,  make  ratings  of 
the  reactions  using  the  boxed  checked,  the  items  listed  below  each 
question,  and  the  criteria  below  as  the  basis  for  your  judgments.  Write 
the  ratings  directly  on  each  worksheet. 

Note  that  in  this  step  and  in  Step  E2.2  the  words  above  the  response 
boxes  on  the  interview  forms  are  not  the  same  as  those  in  the 
guidance  below  and  on  the  questionnaires.  The  boxes  which  hold  the 
same  positions  as  those  on  the  questionnaires,  however,  should  be 
treated  as  if  they  were  labelled  respectively:  strongly  disagree, 
disgaree,  neutral,  agree,  and  strongly  agree  (with  reference  to  the 
guidance  in  these  steps). 

Provide  one  rating  for  Questions  1,  2,  and  3  with  respect  to  how 
motivated  you  think  the  student  is  and  how  that  affected  his 
performance: 

•  If  he  generally  agrees  or  strongly  agrees  on  Quesitons  1  and  2 
and  disagrees  or  strongly  disagrees  on  Question  3,  rate  1. 

•  If  he  is  neutral  or  disagrees  on  Questions  1  and  2  and  is  neutral 
or  agrees  on  Question  3,  rate  2. 

•  If  he  strongly  disagrees  on  Questions  1  and  2,  strongly  agrees  on 
Question  3,  and  gives  good  reasons  for  it,  rate  3. 

•  If  the  responses  strongly  contradict  each  other,  consider  throwing 
out  the  data  for  that  trainee  as  invalid. 

•  If  the  responses  do  not  match  any  of  the  above  situations,  decide 
which  of  the  patterns  most  dosely  matches  and  use  that  rating. 

Provide  one  rating  for  Question  *  based  on  the  combined  effect  you 
think  any  items  listed  there  might  have  had  on  that  student’s 
performance.  If  you  can,  relate  items  listed  to  the  Master  List  of 
Evaluation  Questions  and  review  the  guidance  for  the  applicable 
questions. 

•  If  the  student  checked  strongly  disagree,  disagree,  or  neutral  and 
nothing  is  listed,  rate  1. 

a  If  he  checked  neutral,  agree  or  strongly  agree  and  the  TEE 
question's  guidance  points  to  a  rating  of  2  or  the  items  listed 
taken  together  seem  to  fit  that  category,  rate  2. 

•  If  he  checked  agree  or  strongly  agree  and  nothing  is  listed, 
rate  2. 

•  If  he  checked  agree  or  strongly  agree  and  the  items  listed  justify 
it,  rate  3. 
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E2.2  Locate  any  instructor  reaction  instruments  (Worksheet  C1.12  or  CM3) 

filled  out  in  the  process  evaluation.  For  each  instructor,  make  ratings 
of  the  reactions  using  the  boxes  checked  for  each  task  listed  below 
each  question.  If  no  tasks  arc  listed,  make  no  rating,  unless  it  is  clear 
that  the  instrument  was  given  after  a  training  event  that  applied  to 
only  one  task. 

Rate  each  of  questions  1  through  5  as  follows  for  each  task  listed: 


IF  THE  INSTRUCTOR  CHECKED:  THEN  RATE: 

Strongly  agree  or  1 

Agree 

Neutral  or  2 

Disagree 

Strongly  disagree  3 


E2.3  Go  through  all  of  the  above  mentioned  trainee  and  instructor  reaction 

worksheets  and  all  of  the  product  and  process  data  collection 
worksheets  and  summarize  the  ratings  for  each  question  in  each 
training  event  that  had  two  or  more  ratings  by  different  individuals. 

Do  this  by  averaging  the  ratings  and  rounding  to  the  nearest  whole 
rating.  If  the  average  is  exactly  "1.5"  or  "2.5,"  consider  the 
information  in  the  description  column  in  deciding  whether  to  round  up 
or  down.  If  there  is  nothing  written  there,  make  the  decision  randomly 
so  that  you  round  up  as  often  as  you  round  down. 

Write  the  average  ratings  on  blank  copies  of  the  appropriate 
worksheets.  Fill  in  as  much  of  the  heading  as  necessary  for  quick 
reference  and  keep  it  with  the  corresponding  worksheets  that  wore 
summarized. 

In  summarizing  data  in  the  next  two  steps,  you  must  distinguish 
between  product  and  process  data.  The  questions  on  the  product  and 
process  worksheets  are  automatically  separated  by  product  and  process; 
however,  the  questions  on  trainee  and  instructor  reaction  instruments 
may  not  he.  IJse  the  foliowing  guidance  to  make  that  distinction: 

•  On  the  trainee  reaction  instrument: 

—  Questions  1  through  3  should  be  considered  process  questions. 

—  Question  4  can  be  cither  product,  process,  or  both  depending 
on  the  response. 

•  On  the  instructor  reaction  instrument: 

—  Questions  1  through  4  are  product  questions. 

—  Question  5  can  be  either  or  both  depending  on  the  responses. 
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•  Label  questions  (from  the  Master  List)  on  the  trainee  and 
instructor  reaction  instruments. 

—  If  they  evaluate  training  materials,  label  them  product, 

—  If  they  evaluate  the  process  of  training,  label  them  process. 

Guidelines  for  Worksheets  E2.I  and  E2.2 

E2.4  In  this  step,  the  rating  data  for  the  tests  are  summarized,  i.e.,  data 

from  Master  List  Questions  1  through  26.  You  should  perform  this  task 
first  for  product  evaluation  data  recorded  on  copies  of  Worksheet  B2, 
and  again  for  process  evaluation  data  recorded  on  Phase  C  worksheets 
which  apply  to  testing  events. 

Fill  in  the  headings  to  Worksheets  E2.1  and  E2.2.  Make  one  copy  of 
Worksheet  E2.1  for  each  task  TLO  and/or  LO  evaluated  and  enter  the 
test  item'  number  of  the  items  evaluated. 

Using  the  summary  ratings  from  Step  E2.3  above,  or  raw  data  where 
only  one  rater  was  involved,  go  through  the  worksheets  and  record  the 
Master  List  question  number  and  the  rating  for  each  rating  of  2  or  3 
on  the  sheet  for  the  appropriate  task.  Do  not  record  "1"  ratings. 

Note  that  the  left  side  of  Worksheets  E2.1  and  E2.2  is  for  product 
data;  the  right  side  is  for  process  data. 

Record  ratings  from  questions  which  apply  to  test  items  under  the 
appropriate  item  numbers,  and  to  task/TLOs  or  LOs  in  the  space  at  the 
bottom  of  Worksheet  E2.1.  Record  ratings  from  TEE  questions  that 
apply  to  tests  as  a  whole  on  Worksheet  E2.2. 

Guidelines  for  Worksheet  E2.3 

E2.5  In  this  step,  the  rating  data  from  Master  List  Questions  27  through  93 

(the  presentation  section)  are  summarized.  As  in  the  previous  step,  you 
should  perform  this  task  first  for  product  evaluation  data  and  again  for 
process  data. 

Fill  in  the  heading  to  Worksheet  E2.3,  Make  one  copy  for  each 
task/TLO  and/or  LO  evaluated. 

Again,  use  the  summary  ratings  from  Step  E2.3,  except  in  places  where 
only  one  rater  was  involved.  Go  through  the  applicable  worksheets  and 
record  the  Master  Line  question  number  and  the  rating  for  each  rating 
of  W2M  or  M3M  on  the  sheet  for  the  appropriate  task.  Do  not  record  "I" 
ratings.  Record  ratings  related  to  presentation  components,  to  a  scries 
of  presentation  components,  to  training  events,  to  LOs,  or  task/TLOs  on 
the  top  part  of  the  worksheet  under  the  heading  to  that  effect. 

Record  ratings  applicable  to  lessons,  sets  of  LOs  or  tasks,  or  to  the 
course  on  the  bottom  part  of  the  worksheet  on  the  sheet  for  each 
applicable  task. 

Enter  the  instructor  code(s)  for  the  primary  ins  true  tor  (s)  for  each  task 
In  the  blank  at  the  bottom  of  tlie  worksheet. 

Examples  of  Task  F2  worksheets  are  found  on  the  following  pages. 
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Date:  L£i  bJsLzL. 


Evaluator:  _ 

T. 

\Te 

Course:  / & 

£ 

A  i~ 

Test:  F~ t  / 

i 

i 

WORKSHEET  E2U:  SUMMARY  OF  TEST  ADEQUACY  DATA  FOR  OBJECTIVES 


Task/TLO  or  LO:  7 


Rage _ L.  of  L  for  this  objective 


PRODUCT 

PROCESS 

Questions  by  test  item: 

Questions  by  test  item: 

item 

item  #  S7 , 

item  #  _ 

item  #  7 

item  //  _  jf-_ 

item  0 

5-3 

S-2 

/f -a 

/f-A 

C>-1 

/y-: 

rt-Z 

H-  A 

- 

item  # _ 

item  # _ 

item  £ _ 

item  * 

item  0 _ 

item  ^ 

* 

item  £ _ 

item  6 

item  # _ 

item  £ _ 

item  #  , 

item  # 

Questions  by  objective  (TLO  or  LO): 

/-A 

a-j 

Questions  by  obj#x3ive  (TLO  or  LO): 

Dates  /$‘  Hi  $Ll 

Evaluators  V  -  'T e.  *  s  c  n 
Course:  /6  J5  A  ±  T 
Test:  F  waJ _ 


WORKSHEET  E2A  SUMMARY  OF  TEST  ADEQUACY  DATA  FOR  THE  TEST 


PRODUCT 

PROCESS 

a/  -  J 

<2  4*3 

• 

• 
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Date:  A3  Jci  w  * 
Evaluator:  .J" ^  _  CT*?  n  $  ('ai 
Course:  /  &  S  Alt — 
Lesson: _ ^  oatj/rio 


WORKSHEET  E2.3:  SUMMARY  OF  PRESENTATION  DATA 


TASK/TLO  or  LOi 

PRODUCT 

PROCESS 

Questions  by  presentation  components, 

Questions  by  presentation  component, 

series  of  ccmponents,  LO,  or  task/TLO: 

series  of  components,  training  event, 

LO,  or  task/TLO: 

3S-ji 

42  *3 

VJ  -3 

?6-3 

Questions  by  lesson  (each  one  that 

Questions  by  lesson  (each  one  that 

applies  to  this  LO  or  task/TLO,  by 

applies  to  this  LO  or  task/TLO,  by 

set  of  LOs  or  tasks/TLOs  (of  which  this 

set  of  LOs  or  tasks/TLOs  (of  which  this 

LO  or  task/TLO  is  one),  or  by  course: 

LO  or  task/TLO  is  one),  or  by  course: 

3  3  -A 

3  5  -i 

36-  * 

9/  -2 

9 

Instructor  Code: 


m at irtLfc atejw  <r,  f^r.  «*,  f.  f,  r*^A.<,r 


Task  E3:  Identify  task  and  team  function  performance  discrepancies. 

Once  performance  data  and  training  observations  have  been  summarized 
the  next  step  is  to  analyze  them  to  see  if  there  are  any 
PERFORMANCE  DISCREPANCIES.  A  discrepancy  occurs  whenever 
there  is  difference  between  the  trainees'  expected  performance  and 
what  they  actually  did  in  the  test  situation.  Identification  of  these 
discrepancies  and  the  summary  of  a  few  other  types  of  relevant  data 
are  the  purposes  of  this  task. 

An  important  general  consideration  is  the  selection  of  a  minimum 
standard  for  use  in  defining  task  discrepancies.  Such  a  standard  is  the 
percentage  of  trainees  permitted  to  fail  while  still  considering  task 
performance  acceptable.  You  and  your  organization  need  to  decide 
what  the  minimum  standard  is  for  each  task.  However,  the  following 
is  recommended  and  is  generally  accepted  in  Army  training: 

Suggested  Standard:  Less  than  20%  of  the  crews/ 

individuals  receive  "NO  GO" 
in  the  first  test  trial 

This  task  will  be  conducted  for  tasks,  team  functions,  or  both.  If  you 
have  both,  conduct  the  task  once  for  tasks  and  once  for  team 
functions. 

Guidelines  for  Worksheet  E3.1 

E3.1  On  V/orksheet  E3.1,  enter  the  task  numbers  and  "%  NO  GO"  data  from 

Worksheet  El  for  each  task  being  evaluated.  If  you  have  both  final 
exam  and  within-coursc  test  data,  enter  It  for  both.  If  there  is  only  a 
final  exam,  use  only  the  columns  under  the  final  exam  heading. 

Look  at  copies  of  Worksheets  E2.1  and  E2.2  and  make  a  Judgment  on 
test  adequacy  for  each  task.  Enter  it  in  the  columns  so  labeled  for 
the  final  and  any  within-coursc  test  according  to  the  guidelines  in  the 
table  below.  Also,  drde  serious  deficiencies  on  Worksheets  E2.1  and 
E2.2.  Note  that  the  deficiencies  noted  for  the  test  as  a  whole  apply 
to  each  and  every  task.  Also,  when  considering  the  "seriousness"  of 
deficiencies,  take  into  account  whether  It  applies  to  the  whole  task  or 
only  Influences  certain  test  items  or  LOs,  and  if  so,  Ik>w  many  items 
or  LOs  In  the  task  are  affected. 


IF? 

THEN  ENTER: 

There  are  no  deficiencies  or  not  many  deficiencies,  and 
any  deficiencies  shown  do  not  appear  to  be  serious* 

/ 

Tltcre  arc  numerous  deficiencies  listed  and/or  one 
or  more  deficiencies  appear  to  be  serious* 

X 

The  deficiencies  listed  do  not  clearly  fit  cither 
of  the  above  descriptions 

....  .  . — - - — - 

? 

* Refer  to  Appendix  G  to  ascertain  the  seriousness  of  deficiencies.  It 
contains  guidance  for  each  Master  List  question. 
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The  three  marks  shown  above  have  the  following  meanings  which  you 
should  also  keep  in  mind  as  you  judge  the  test  for  each  task.  They 
will  also  apply  to  judgments  of  the  training  for  each  task  in  later  steps 
(judged  either  by  performance  scores  or  observations  of  the 
presentation). 

y  -  The  test  or  training  is  probably  adequate.  From  the 

observations  made,  there  is  nothing  to  indicate  that  the 
test  is  invalid  or  that  the  training  is  discrepant. 

X  -  The  test  or  training  is  probably  inadequate.  From  the 

observations  made,  there  is  enough  evidence  to  indicate 
that  the  test  is  probably  invalid  or  that  a  training 
discrepancy  probably  exists. 

?  -  It  is  not  known  whether  the  test  or  training  is  adequate  or 

not. 

•  In  the  case  of  a  test,  there  are  some  deficiencies, 
but  it  is  not  clear  whether  they  are  serious  enough 
when  taken  together  to  make  the  test  invalid. 

•  In  the  case  of  training,  either  the  test  Is  Inadequate 
or  questionable  (when  performance  scores  are 
judged)  or  there  are  some  presentation  deficiencies, 
but  it  is  not  clear  whether  they  are  serious  enough 
to  cause  a  training  discrepancy  (when  presentation 
ratings  are  judged). 

E3.2  Go  down  the  "Task"  column  for  the  final  and  any  within-course  test 

and  record  whether  the  performance  scores  In  each  indicate  task 
discrepancies  or  not.  Use  the  following  guidelines.  Note  that  they 
assume  a  2093  standard  and  should  be  adjusted  accordingly  if  you  use 
another  standard. 


IF  TEST  ADEQUACY  IS: 

AND  %  NO  GO  IS: 

THEN  RECORD: 

y 

0  TO  19% 

y 

y 

20  TO  1009$ 

X 

?  or  X 

— 

7 

E3.3  Go  down  the  "Combined  Task  Rating**  column  and  record  a  combined 

rating  for  each  task.  If  there  is  no  within-course  data  separate  fiom 
linal  exam  data,  skip  this  step  and  use  the  final  exam  task  ratings 
instead  of  a  combined  rating  later  on. 


Use  the  following  guidelines  to  ascertain  the  combined  task  ratings: 


I  OS 


IF  FINAL  EXAM 
TASK  COLUMN  IS: 

AND  WITH  IN-COURSE 
TEST  TASK  COLUMN  IS: 

THEN  RECORD: 

/ 

V,X,  or? 

/ 

X 

✓ 

? 

X  or  ? 

X 

? 

/,  X,  or  ? 

Same  as  within 
course  test 

E3.4  Look  over  copies  of  Worksheet  E2.3  and  make  a  presentation  rating  in 

the  "Presentation  Rating”  column  of  Worksheet  E3.1  for  each  task  using 
the  guidelines  below.  At  the  same  time  circle  serious  deficiencies  on 
Worksheet  E2.3  for  future  reference. 


IF: 

THEN  ENTER: 

There  are  no  deficiencies  or  not  many  deficiencies, 
and  any  deficiencies  shown  do  not  appear 
to  be  serious* 

/ 

Thare  are  numerous  deficiencies  listed  and/or 
one  cr  more  deficiencies  appear  to  be  serious* 

X 

The  deficiencies  listed  do  not  clearly  fit 
either  of  the  above  descriptions. 

7 

•Refer  to  the  guidance  in  Appcndir.  C  to  ascertain  seriousness  for  each 
Master  List  question. 


Again,  you  should  keep  in  mind  the  meaning  of  each  mark  as  you  make 
these  judgments: 

V  -  The  training  is  probably  adequate.  From  the  observations  made, 
there  is  nothing  to  indicate  that  the  training  is  discrepant. 

X  -  The  training  is  probably  Inadequate.  From  the  observations 
made,  there  is  enough  evidence  to  indicate  that  a  training 
discrepancy  probably  exists. 

?  -  It  is  not  known  whether  the  training  is  adequate  or  not.  There 
are  some  presentation  deficiencies,  but  it  is  not  clear  whether 
they  arc  serious  enough  to  cause  a  training  discrepancy. 

Note  that  these  presentation  ratings  are  less  accurate  than  performance 
scores  on  an  adequate  test,  which  fact  is  taken  into  account  in  the 
next  step. 
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E3.5  When  you  have  combined  task  ratings  from  within  course  tests  (or  task 

ratings  from  a  final  exam  alone)  and  presentation  ratings,  combine  the 
two  into  a  final  task  using  the  guidelines  below.  Enter  the  rating  in 
the  last  column  of  Worksheet  E3.1. 


IF  COMBINED  TASK 
RATING  IS: 

AND  PRESENTATION 
RATING  IS: 

THEN  RECORD: 

/ 

V  or  ? 

7 

X 

?* 

X 

/ 

?* 

X  or  ? 

X 

7 

7  1 

7 

X 

V 

A 

? 

7* 

*In  these  cases  you  should  examine  your  evaluation  methods  and/or  the 
training  and  try  to  explain  why  there  is  a  discrepancy  between  the 
performance  scores  and  the  presentation  ’  ratings.  Look  for  reasons 
why  the  test  adequacy  and  presentation  ratings  might  be  wrong.  If 
you  find  errors,  go  back  through  the  appropriate  data  collection  and 
data  summary  worksheets  and  make  corrections.  Finally,  enter  the 
correct  rating  here. 

Guidelines  for  Worksheet  E3.3 

E3.6  If  related  crew/team  and  individual  performances  were  both  scored  or 

if  individual  tasks  can  be  clustered  into  superordinate  functions, 
summarize  the  data  further  by  team  function  or  by  superordinate 
functions  for  groups  of  individual  tasks. 

Locate  Worksheet  E3.2.  On  the  top  line  in  Box  PI  of  separate  copies 
of  the  worksheet,  list  the  function  numbers  for  the  team  or 
superordinate  individual  functions  evaluated  in  the  TEE.  if  you  are 
clustering  tasks  into  individually  performed  or  other  sub-team  functions, 
you  may  need  to  define  those  functions,  giving  them  names  (actions) 
and  numbers,  if  that  has  not  been  done. 

Below  Box  P\  on  each  worksheet,  list  the  subordinate  functions,  tasks, 
and  any  communication  skills  hierarchically,  one  in  each  box.  In  this 
hierarchical  listing,  the  functions,  tasks,  or  skills  on  the  row  below  for 
any  box  should  be  such  that  their  successful  performance  is  required 
before  the  task  or  function  in  the  box  above  them  can  be  performed 
acceptably.  Do  not  list  tasks  at  any  given  point  on  the  worksheet  that 
do  not  pass  this  test. 
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You  may  not  need  to  use  ali  of  the  boxes  on  any  worksheet.  On  the 
other  hand,  you  may  need  to  use  a  number  of  worksheets  for  some 
functions.  To  continue  listing  tasks  downward,  list  the  bottom  box 
numbers  and  the  sheet  number  in  the  top  (reference)  box  on  additional 
worksheets.  Also  list  the  sheet  numbers  of  the  additional  worksheets  in 
the  reference  boxes  on  the  bottom  row  of  the  sheet  from  which  they 
are  continued. 

To  expand  horizontally,  reference  additional  worksheets  in  the  reference 
boxes  provided  on  the  upper  and  lower  right.  On  the  additional  sheets, 
reference  the  correct  superordinate  box  in  the  reference  box  at  the 
top.  You  can  cross  out  Box  //I  and  use  the  next  row  if  there  is  more 
than  one  additional  task.  You  may  wish  to  tie  all  of  the  worksheets 
together  with  one  sheet  showing  the  top-level  functions  taught  in  the 
course. 

On  the  second  line  of  each  box,  list  the  final  rating  (i.e.,  Y ,  X,  or  ?) 
for  each  task,  individual  function,  team  function,  and  any 
communication/coordination  skills  from  the  right-hand  column  of 
Worksheet  E3.1. 

In  the  blank  to  the  right  of  each  box  which  has  subordinate  functions, 
tasks,  or  skills  below  it,  rate  the  relationship  of  the  performances 
shown  in  the  superordinate  box  to  those  in  The  subordinate  boxes  below 
it.  Use  the  [  idance  in  the  table  below  in  doing  so.  The  meaning  of 
these  ratings  is  explained  following  the  table. 
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IF  SUPERORDINATE 
BOX  IS: 

AND  SUBORDINATE  BOXES 
HAVE: 

THEN  RATE  THE 
RELATIONSHIP; 

y 

1/2  or  more  >/fs  and  no  X's 

OK 

One  or  more  X's 

> 

f 

More  than  1/2  ?'s 

7 

X 

An  y  <s 

- 7 

\ 

One  or  more  X's 

X 

One  or  more  ?'$  and  no  X's 

7 

? 

More  than  1/2  X'i 

«  1 

X 

Other  sets  of  ratings 

7 

♦ 

ill 


«  ^  « 
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Key  to  Ratings  of  Hierarchical  Performance  Relationships! 

OK  -  Performance  on  the  superordinate  function  and  subordinate 
functions  or  tasks  is  acceptable. 

Performance  on  the  superordinate  function  is  discrepant, 
while  performance  on  all  subordinate  functions  or  tasks  is 
acceptable.  Some  unmeasured  communication/coordinate 
skill  or  other  aspect  of  the  function  which  involves  more 
skill  than  the  simple  sum  of  the  task  performances  is 
discrepant. 

Performance  on  the  superordinate  function  is  acceptable, 
while  performance  on  one  or  more  subordinate  functions  or 
tasks  is  discrepant.  Some  individual  or  event  made  up  for 
the  discrepant  task  perform  ance(s). 

X  -  Performance  on  the  superordinate  function  is  discrepant 

along  with  one  or  more  subordinate  functions  or  tasks.  It 
is  unknown  whether  the  discrepancy  on  the  superordinate 
function  is  due  to  the  discrepancies  on  the  subordinate 
function(s)  or  task(s)  alone,  or  whether  some  skill  involved 
in  the  superordinate  function  is  also  discrepant. 

?  -  Performance  on  the  superordinate  function  and/or  enough 

of  the  subordinate  functions  or  tasks  is  unknown  such  that 
the  nature  of  the  hierarchical  performance  relationship  is 
unknown. 

Examples  of  Task  E3  worksheets  are  shown  on  the  following  pages. 


Date:  /O  Jet  n  .  *?  / 

Evaluator:  _  vJ.v 
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WORKSHEET  E3.1:  IDENTIFY  TASK  PERFORMANCE  DISCREPANCIES 


Task  # 

% 

No  Go 

Test 

Adequacy 

Task 

% 

No  Go 

■ 
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Task 

Combined 

Task 

Rating 

Presenta¬ 

tion 

Rating 

Final 

Task 

Rating 

ra 

33 

i/ 

a 

V 

X 

X 
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X 
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0 

X 

H 

/o 

• 

? 

# 

? 

* 

X 

X 

/<? 

17 

mm 

? 

• 

/j. 

/ 

X 

X 

? 

X 

/r 

m 

X 

K9 

6? 

? 

♦ 

? 

* 

X 

X 

mm 

<2  / 

o 

X 

1  / 

0 

x 

/ 

X 

X 
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WORKSHEET  E3.2:  DATA  SUMMARY  BY  TRAINEE 
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WORKSHEET  E3-3  DATA  SUMMARY  BY  FUNCTION 


WORKSHEET  E3.3  DATA  SUMMARY  BY  FUNCTION 
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PHASE  Fs  DOCUMENT  THE  TEE 


Overview  of  Phase  F 

The  purpose  of  Phase  F  is  to  summarize  the  findings  of  your  work, 
prepare  a  report  documenting  the  effort,  and  organize  the  raw 
worksheets  for  future  use* 

There  are  no  formal  worksheets  accompanying  these  guidelines  for 
Phase  F.  Blank  sheets  of  paper  are  your  '’worksheets." 

Guidelines  for  Phase  F 

Task  FI:  Prepare  report  summarizing  TEE  conduct  and  findings. 

Fl.l  Before  interpreting  the  TEE  results,  look  for  gaps  in  your  data. 

Estimate  what  data  should  have  been  collected  and  were  not.  Decide 
if  it  is  feasible  to  collect  additional  data  before  preparing  and 
submitting  the  report. 

Interpret  the  data  summaries  in  Worksheets  E3.1  and  E3.3.  Make  notes 
on  the  conclusions  you  reach  for  inclusion  in  your  report. 

Examine  Worksheet  E3.1  considering  the  following  points: 

•  In  general,  are  the  tests  valid  and  reliable? 

•  How  Is  the  training  for  the  tasks  or  team  functions  in  the 
course?  How  many  are  acceptable,  marginal,  or  discrepant? 

•  Based  on  the  adequacy  of  the  tests  and  training  presentation,  are 
there  enough  problems  to  warrant  revising  one  or  both  of  them? 
Is  the  course  so  bad  that  a  second  TEE  would  be  in  order 
following  extensive  revisions? 

Look  over  your  copies  of  Worksheets  E2.1  and  E2.2  for  each  lesson: 

•  Which  lessons  need  the  most  work  In  order  to  be  made 
acceptable? 

•  Do  you  see  any  relationship  between  the  worst  lessons  and 
individual  instructor  codes,  or  between  any  such  codes  and 
instructor  deficiencies  rated  on  Worksheet  D3.2? 

For  Worksheet  E3.3: 

•  In  general,  does  the  training  for  team  functions  need  to  be 
improved  or  do  most  team  function  discrepancies  stem  only  from 
failures  on  their  subordinate  tasks'*  If  not,  are  the  measures  of 
com  muni  Celt  ion  and  coordination  skills  adequate? 
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FI. 2  In  writing  your  report,  you  will  want  to  present  some  of  the  data  in 

graphic  or  tabular  formats.  Shown  below  are  some  examples  of  ways 
you  can  display  the  data. 

To  show  the  contrast  among  performance  scores  or  test  adequacy 
ratings  for  tasks  or  team  functions  or  the  final  ratings  for  tasks,  the 
following  types  of  graphs  can  be  used: 


PERFORMANCE  SCORES  ON  TASKS 


COURSE  OUTLOOK  SHOWING  FREQUENCY  OF  TASK  RATINGS. 


S 

L 


A  useful  format  for  displaying  performance  relationships  between  team 
functions  and  subordinate  tasks  or  communication  and  coordination 
skills!  or  for  tasks  and  subordinate  LOs  is: 


PERFORMANCE  SCORE  HIERARCHY  SHOWING  %  NO  GO  AND  SKILL  RATINGS. 


It  is  important  to  present  data  in  numerical  form.  Tables  are  used  for 
that  purpose.  For  example: 


EVALUATION  DATA  BY  TASK 

TASK 

%  NO  GO 

TEST 

ADEQUACY 

PRESENTATION 

RATING 

FINAL  TASK 
RATING 

1 

33 

X 

? 

DISCREPANT 

3 

20 

? 

y 

ACCEPTABLE 

5 

16 

y 

7 

UNKNOWN 

*ji 

0 

7 

X 

UNKNOWN 

9 

25 

y 

? 

DISCREPANT 

14 

22 

? 

y 

ACCEPTABLE 
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The  report  should  contain  these  major  sections: 


a.  Review  of  the  request  for  a  TEE,  and  a  review  of  any  findings 
of  any  previous  TEEs  performed  on  the  course. 

b.  Brief  presentation  of  the  TEE  process,  including  the  general  TEE 
purposes  selected  in  Task  A1  as  reasoas  for  the  approach  taken. 

c.  Brief  summary  of  the  steps  you  took  in  performing  the  TEE, 
noting  any'  reasons  why  you  deviated  from  the  usual/expected 
approach. 

d.  Presentation  of  the  basic  facts  about  the  course  Including: 

a  Number  trained 

•  Number  of  instructors 

•  Number  of  classroom  hours 

q  Goal  of  the  course 

a  Any  special  conditions  of  the  course 

e.  Presentation  ox  performance  data 

•  Number  of  lst-time  "Go's"  on  final  and  within-course  tests 
by  task 

•  Discussion  of  the  adequacy  of  the  test(s) 

f.  Discussion  of  each  task  and  team  function  that  showed  below- 
standard  performance  (those  rated  discrepant  and  perhaps  some 
related  unknown). 

•  General  reasons  why  the  task  or  team  function  failed 

•  Amount  of  effort  required  to  revise  instruction  for  each 
task,  team  function,  or  lesson 

g.  Final  recommendations 

•  Should  an  effort  be  made  to  revise  the  course? 

•  Your  overall  evaluation  of  the  course 

•  Others 

Task  F2:  Preserve  TEE  documentation* 

For  two  reasons,  the  course  materials,  the  product  and  process  data 
collection  worksheets,  summary  worksheets,  the  final  report,  and  other 
TEE  documentation  stould  be  properly  organized  and  stored  where  they 
can  be  accessed  for  future  use.  First,  if  the  course  is  to  be  revised  in 
any  way,  these  TEE  documents  will  identify  the  most  serious  problems 
and  lead  developers  to  the  revisions  that  need  to  be  made.  Second,  If 
another  TEE  is  ever  conducted  for  the  course,  the  documentation  of 
this  TEE  will  be  a  valuable  resource. 

You  should  file  TEE  documents  and  worksheets  by  TEE  task  in  clearly 
marked  folders.  You  will  want  to  have  separate  files  for  many  of  the 
TEE  steps. 
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MASTER  LIST  OF  EVALUATION  QUESTIONS 
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Appendix  A 

MASTER  LIST  OF  EVALUATION  QUESTIONS 
TESTING  QUESTIONS 


Test  Consistency  with  Objectives 

1. 

_ RATING:  (by  objective— TLO  and  LOs): 


ANALYST  ONLY 

Is  the  test  FORMAT  appropriate  to  the  TASK  LEVEL  and  CONTENT  TYPE  of 
the  OBJECTIVE?  Appropriate  formats  are  shown  in  the  table  below. 


And  Test  Format  Is: 

Then  Rate 

short  answer, 
fiil-in  or 
listing 

l 

performance  (for  a 
later  USE-UNAIDED 
objective) 

2 

performance  (for  a 
later  USE-AIDED 

3 

objective) 

If  Task  Level  Is: 
REMEMBER 


And  Content  Type  Is: 

FACT  for  RECALL, 
CONCEPT,  PROCEDURE, 
RULE,  or  PRINCIPLE 


FACTS  FOR 
RECOGNITION 


matching, 
true-false,  or 
multiple  choice 


matchin 

true-false,  or 
multiple  choice 


performance 


short  answer, 
fill-in,  or 
listing 


USE-UNAJOED 

or 

USE-AIDED 


CONCEPT 


performance, 
matching, 
true-false, 
multiple  choice, 
short  answer,  or 
fill-in 


listing 


PRINCIPLE,  RULE, 
or  PROCEDURE 


performance, 
true-false 
multiple  choice, 
short  answer,  or 
fill-in 


listing  or 
matching 


2 


EXAMPLE: 


OBJECTIVE:  "The  student  will  state  the  rule  lor  finding  total  inductance  on  a 
series  circuit." 

TEST  ITEM:  "Which  of  the  following  is  the  correct  rule  for  finding  total 
inductance  in  a  series  circuit? 

A.  E  =  IR  C.  R  =  IE 

B.  RLC  =  Z  D.  None  of  the  above 

In  this  example,  the  objective  is  at  the  Remember  task  level  and  content 
type  is  Rule.  The  test  item  format  is  multiple  choice.  Looking  across  the 
rating  table  for  Q.l,  we  see  that  this  combination  requires  a  rating  of  "2." 

(It  is  obvious  why  this  format  is  bad;  the  multiple  choice  format  changes  the 
conditions  and  action.) 

ANALYST  ONLY 

2.  Are  there  test  items  for  the  TLO  or  all  of  its  critical  parts /LOs? 

NOTE:  As  defined  in  Section  4  of  the  Guidelines  for  evaluating 

objectives  in  Task  A4,  critical  parts/LOs  have  the  following 
characteristics: 

•  They  are  some  of  the  most  measurable  and  observable 
elements  of  the  task. 

•  They  have  serious  consequences  of  inadequate  performance. 

•  They  are  common  sources  of  failure. 

•  They  are  actions  which  the  soldiers  of  the  target 
population  do  not  already  know  how  to  do  and  must 
therefore  be  taught. 

RATING:  If  there  are  test  items  for  the  TLO  or  all  of  its  critical 

parts/LOs,  rate  1. 

If  there  are  no  test  items  for  the  TLO  and  for  some  critical 
parts/LOs,  rate  2. 

If  there  are  no  test  items  for  the  TLO  and  for  most  of  the 
critical  parts/LOs,  rate  3. 

NOTE:  For  entry  tests,  treat  the  top  level  of  entry  skills  as  TLOs. 
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3.  Is  there  a  test  item  for  each  critical  part  of  each  LO? 

The  critical  parts  of  LOs  are  defined  below. 

For  the  REMEMBER  TASK  LEVEL  each  piece  of  information  is  critical.  Each 
of  the  following  should  be  tested: 

•  Facts 

•  Concept  characteristics 

•  Procedural  steps 

•  Steps  and  conditions  for  branching  in  a  rule 

•  Guidelines  in  a  principle 

For  the  USE-AIDED  and  USE-UNAIDED  TASK  LEVELS,  the  parts  of  the 
OBJECTIVE’S  ACTION  are  critical: 

•  For  CONCEPTS,  are  all  CRITICAL  CHARACTERISTICS  tested? 

Can  you  be  sure  the  student  knows  each  one  and  can  you 
identify  which  ones  he  does  not  know? 

•  For  PROCEDURES,  is  each  step  performed? 

•  For  RULES  and  PRINCIPLES,  is  each  step  performed?  Are  there 
items  for  all  possible  branches  or  decisions,  or  at  least  those 
that  would  commonly  occur  on  the  job? 

RATING:  If  there  are  test  items  for  all  critical  parts,  rate  the  LO  1. 

If  there  are  test  items  for  many  but  not  all  critical  parts,  rate 
the  LO  2* 

If  there  are  test  item*  for  only  a  few  critical  parts  or  for  no 
parts,  rate  the  LO  3. 


ANALYST  ONLY 

4.  Does  the  TASK  LEVEL  of  the  test  item  match  the  TASK  LEVEL  of  its 
OBJECTIVE? 

The  task  level  of  the  test  item  should  match  the  task  level  of  the  objective. 
This  means  that  the  action  verb  in  the  test  item  should  be  the  same  as  the 
action  verb  in  the  objective,  or  at  least  the  same  behavior  must  be  required. 
If  it  isn’t,  the  test  item  is  measuring  something  different  from  what  was 
required  In  the  objective. 

Compare  the  ACTIONS  of  the  test  item  and  the  OBJECTIVE.  Record  the 
corresponding  rating  indicated  in  the  following  table. 

RATING  (by  item): 
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Test  Item  TASK  LEVEL 


OBJECTIVE 

TASK 

LEVEL 


• 

REMEMBER 

USE-UNAIDED 

USE-AIDED 

REMEMBER 

i 

2 

2 

USE-UNAIDED 

2 

1 

3 

USE-AIDED 

3 

3 

1 

EXAMPLE: 

OBJECTIVE:  "The  student  will  Write  from  memory  the  steps  of  the  procedure 
for  field  stripping  the  SGT  YORK  cannon.” 

TEST  ITEM:  "Using  the  SGT  YORK-10  manual,  field  strip  the  SGT  YORK 
cannon.” 

In  this  example,  the  objective  task  level  is  Remember.  The  test  item  task 
level  is  Use-Aided.  Referring  to  the  rating  table  for  Q.4,  you  would  rate  this 
combination  "3.” 


ANALYST  ONLY 

5.  Does  the  content  of  the  test  item  match  the  content  of  its  OBJECTIVE7 

For  the  REMEMBER  TASK  LEVEL  the  content  of  the  items  and  the 
OBJECTIVE  is  the  same  when  the  items  test  the  identical  information 
specified  by  the  OBJECTIVE. 

For  the  USE-AIDED  and  USE-UNAIDED  TASK  LEVELS,  the  content  of  the 
items  is  the  same  as  the  OBJECTIVE  when  the  ACTION  of  the  items,  or  the 
combined  ACTIONS  of  a  group  of  items,  is  the  same  as  the  OBJECTIVE’S 
ACTION. 

RATING:  If  the  content  Is  the  same,  rate  1. 

If  the  content  is  slightly  different,  rate  2. 

If  the  content  is  very  different,  rate  3. 


EXAMPLE: 

OBJECTIVE:  "Given  a  globe  valve,  rags,  prussian  blue,  gasket  material, 

packing,  and  tools,  the  student  will  disassemble  and  reassemble 
the  globe  valve  (within  30  minutes).” 

TEST  ITEM:  "Which  of  the  following  materials  arc  required  for  disassembling 
and  reassembling  the  globe  valve?" 

A.  rags  C.  gasket  material 

B.  prussian  blue  D.  All  of  the  above 
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For  USE  level  objectives,  the  content  is  the  same  when  the  action  of  the  test 
item  is  the  same  as  the  objective’s  action.  Since  It  is  obvious  that  this  test 
item  does  not  address  the  objective  at  all,  it  should  be  rated  "3." 


6.  Do  the  CONDITIONS  of  the  test  item  match  the  CONDITIONS  of  its 
OBJECTIVE? 

Are  the  CONDITIONS  under  which  the  test  item  is  administered  the  same  as 
those  stated  in  the  OBJECTIVE? 

RATING:  Give  one  rating  for  each  item  within  each  OBJECTIVE  on  each 

test.  Make  notes  in  the  ’’Description”  column  indicating  the 
nature  of  the  discrepancy(ies). 

If  the  item  and  OBJECTIVE  CONDITIONS  match  exactly,  rate 
the  item  1. 

If  there  is  a  minor  mismatch,  rate  it  2. 

If  there  is  a  severe  mismatch,  rate  the  item  3. 

EXAMPLE: 

OBJECTIVE:  ’’Given  any  resistor  with  four  color  bands,  the  student  will  follow 
the  procedure  for  determing  ohmic  value  as  indicated  by  the 
color  bands.” 

TEST  ITEM:  ’’For  each  of  the  resistors  pictured  below,  determine  the  value  in 
ohms,  and  write  the  value  in  the  space  next  to  each  resistor." 

The  objective  says  the  student  will  be  given  resistors,  while  the  test  item 
gives  pictures.  This  might  be  close  enough  if  the  pictures  are  actual  size  and 
in  color.  Since  this  is  a  minor  mismatch,  it  would  be  rated  ”2.” 


7.  Do  the  STANDARDS  of  the  test  item  match  the  STANDARDS  of  its 
OBJECTIVE? 

Are  the  STANDARDS  for  the  individual  test  item  the  same  as  those  stated  in 
the  OBJECTIVE?  (For  many  forms  of  written  tests  given  at  the  REMEMBER 
TASK  LEVEL,  the  STANDARDS  are  based  simply  on  selection  of  the  correct 
answer.  In  such  cases,  this  question  may  not  be  applicable.) 

RATING:  Give  one  rating  for  each  item  within  each  OBJECTIVE.  Make 

notes  in  the  "Description"  column  indicating  the  nature  of  the 
discrepancy(ies). 

If  the  item  and  the  OBJECTIVE'S  STANDARDS  match  ex  tly, 
rate  the  item  1. 

If  there  is  a  minor  mismatch,  rate  it  2. 

If  there  is  a  severe  mismatch,  rate  the  item  3. 
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EXAMPLE: 


OBJECTIVE:  "Given  the  formula  for  Ohm’s  Law,  E=IR,  and  any  two  values 
typical  of  values  encountered  on  the  job,  the  student  will  solve 
for  the  third  value  correct  to  two  decimal  places.” 

TEST  ITEM:  "Ohm’s  Law  is  E=IR.  If  1=200  ma.,  and  R=47K  ohms,  then  E= 


The  item  should  specify  that  accuracy  to  two  decimal  places  is  required. 
Since  this  is  a  minor  mismatch,  it  would  be  rated  ”2.” 


Test  Adequacy 

8.  For  true-false,  multiple  choice,  and  matching  items  is  only  one  answer 
correct? 

For  selected  response  items  (true-false,  multiple  choice,  and  matching)  there 
should  be  only  one  correct  answer: 

RATING:  If  there  is  only  one  correct  answer,  rate  1. 

If  more  than  one  answer  can  be  correct,  rate  3. 

EXAMPLE: 

ITEM:  "If  you  are  driving  down  a  steep  grade,  you  would  be  likely  to  shift 
gears: 

a.  down  for  power. 

b.  up  for  power. 

c.  down  to  decelerate. 

d.  down  to  slow  the  vehicle. 

Choices  "c"  and  "d"  are  paraphrases  of  each  other,  giving  two  possibly  correct 
answers.  Therefore  this  item  should  be  rated  "3.” 

9.  For  short  answer,  fill-in,  listing,  and  performance  items,  are  all  acceptable 
answers  in  the  answer  key? 

For  constructed  response  items  (short  answer,  fill-in,  listing,  and  performance) 
multiple  correct  answers  may  be  acceptable. 

RATING:  If  all  correct  answers  are  in  the  answer  key,  rate  1. 

If  some  correct  answers  are  not  in  the  answer  key,  rate  3. 
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10.  Does  the  test  item  provide  opportunities  for  COMMON  ERRORS  to  be  made? 

Do  the  possible  responses  to  the  item  aliow  students  to  choose  or  commit 
most  errors  common  in  the  real  job  task? 

RATING;  If  yes,  rate  1. 

If  no,  rate  3. 

NOTE:  You  may  need  the  help  of  a  subject  matter  expert  to  identify 

COMMON  ERRORS. 


11.  Is  the  language  of  the  test  item  easy  for  students  to  understand? 

Considering  the  reading  level  and  intelligence  of  the  students,  are  any  long 
sentences  or  difficult  words  used  other  than  job-related  words  taught  in  the 
course. 

RATING:  If  the  language  of  the  item  is  easy  for  the  students,  rate  l. 

If  the  language  is  somewhat  difficult,  rate  2. 

If  the  language  of  the  item  is  very  difficult,  rate  3. 
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12.  Is  the  test  item  different  from  previous  PRACTICE  and  EXAMPLES?  (This 
question  applies  only  to  items  at  the  USE-AIDED  or  USE-UNAIDED  TASK 
LEVELS,  and  the  CONCEPT,  RULE,  or  PRINCIPLE  CONTENT  TYPES.) 

The  test  item  should  not  be  the  same  as  previously  used  EXAMPLES  or 
PRACTICE.  (One  exception  is  that  items  testing  COMMON  ERRORS  may  be 
the  same.) 

RATING:  If  the  test  item  is  different,  rate  l. 

If  test  item  has  been  presented  before  as  an  EXAMPLE  or  in 
PRACTICE,  and  the  TASK  LEVEL  of  the  OBJECTIVE  is  USE- 
AIDED,  rate  2. 

If  the  test  item  has  been  presented  before  as  an  EXAMPLE  or  in 
PRACTICE,  and  the  TASK  LEVEL  of  the  OBJECTIVE  is  USE- 
AIDED,  rate  3. 


A-S 


13*  Is  the  answer  to  the  test  item  given  away  by  another  item  or  group  of  items 
on  the  test? 

RATING:  If  the  answer  is  not  given  away,  rate  1* 

If  other  items  give  clues  to  the  answer,  rate  2. 

If  the  answer  can  be  found  in  other  item(s),  rate  3. 

14.  Is  the  answer  to  the  test  item  dependent  on  answering  a  previous  item  or 
group  of  items  correctly? 

RATING:  If  the  answer  does  not  depend  on  o:her  items,  rate  1. 

If  previous  items  must  be  answered  correctly  in  order  to  answer 
this  item  correctly,  rate  3. 

15*  Are  sketches  and  diagrams  used  in  the  test  item  easy  to  understand? 

RATING:  If  sketches  and  diagrams  are... 

...easy  to  understand,  rate  1. 

...somewhat  confusing,  rate  2. 

••o  very  confusing,  rate  3. 

16.  Is  the  test  item  tricky  or  misleading  in  that  it  points  to  an  incorrect  answer? 

RATING:  If  the  item  is  not  misleading,  rate  1. 

If  it  is  somewhat  misleading,  rate  2. 

If  the  item  is  very  misleading,  rate  3. 


PRODUCT  ONLY 


17.  Is  the  test  item  well-constructed? 

Select  the  appropriate  set  of  criteria  below  depending  upon  the  test  item 
format  being  used.  On  Worksheet  B2  write  the  name  of  the  test  format 
below  the  TEE  question  marker.  Record  it  again  if  the  format  changes. 
Rate  the  Item  as  indicated  below,  noting  the  !etter(s)  corresponding  to  the 
identified  deficiency(ies)  in  the  "Description"  column.  Note  that  critical 
criteria  are  marked  with  asterisks  in  the  supplementary  guidance  to  this 
question. 

RATING:  If  the  item  is  well  constructed,  i.e.,  it  meets  all  the  criteria, 

rate  1. 
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If  the  item  is  deficient  on  one  or  more  non-critical  criteria  (but 
not  on  any  critical  criteria),  rate  2. 

If  the  item  is  deficient  on  one  or  more  critical  criteria,  rate  3. 


CRITERIA  FOR  WELL-CONSTRUCTED  TEST  ITEMS 


•  True-False 

*a.  The  item  should  include  only  one  statement  (idea)  to  be  judged  true  or 
false. 

*b.  Negative  statements  and  double  negatives  should  not  be  used, 

c.  The  item  should  be  short. 

*d.  The  item  should  not  be  so  obvious  that  it  can  be  answered  correctly  on 

the  basis  of  common  sense  alone. 

*e.  Absolutes,  like  "none,”  "never,”  "all,”  and  "every,"  should  not  be  used. 

*f.  The  item  should  not  contain  the  words  "same,"  "any,"  or  "generally." 

*g.  The  item  should  deal  with  things  that  are  clearly  either  true  or  false. 

EXAMPLE: 

T  F  It  is  not  appropriate  to  shift  from  a  higher  gear  to  a  lower  gear 
if  added  power  is  not  necesary. 

Learners  reading  this  item  must  attempt  to  translate  the  statements  into 
positive  form  before  answering  it.  This  violates  criterion  "b"  above.  Since 
the  item  is  deficient  on  one  critical  (*)  criterion,  it  should  oe  rated  ”3." 


•  Multiple  Choice 


a.  All  alternatives  should  have  the  same  grammatical  structure. 

b.  The  item  stem  should  be  worded  positively,  that  is,  words  like  "no"  and 
"not"  should  not  be  used. 

c.  Repetitive  phrases  should  be  placed  in  the  stem,  rather  than  in  the 
alternatives. 

d.  Numerical  choices  should  be  listed  in  order  of  magnitude. 

e.  The  item  should  not  require  a  great  deal  of  reading. 

*f.  Wording  of  the  stem  should  be  clear  and  unambiguous,  so  that  only  the 

one  correct  answer  is  possible. 

*g.  The  article  "a"  or  "an"  should  not  be  at  the  end  of  the  stem. 
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*h.  All  alternatives  should  be  plausible  to  someone  who  doesn't  know  the 
answer. 

i.  Alternatives  like  "A  and  B  only"  or  "all  of  the  above"  should  not  be 
used. 

*j.  Alternatives  should  be  approximately  the  same  length. 

*k.  Extreme  adjectives,  such  as  "complete,"  "total,"  and  "absolute"  should 
be  avoided. 

EXAMPLE: 

ITEM:  The  best  reason  for  avoiding  carbon  tetrachloride  in  cleaning 

tools 

% 

a.  is  because  it  is  very  expensive. 

b.  is  because  it  will  harm  the  tools. 

c.  Is  because  it  will  affect  the  color  of  the  tools. 

d.  is  because  it  is  likely  to  produce  flames,  which  exist  in 

sufficient  amount  to  harm  those  around  It. 

This  item  has  two  deficiencies.  The  phrase  "is  because  it"  belongs  in  the 
stem,  not  in  the  alternatives  (criterion  "c"),  and  alternative  "d"  is  longer  than 
the  other  three  (criterion  "j").  'Test-wise"  students  know  that  a  longer 
answer  is  often  correct  because  the  test  developer  often  spends  more  time 
writing  the  correct  answers  than  the  incorrect  alternatives.  Since  the  item  is 
deficient  on  two  criteria,  one  of  which  is  critical  (*),  it  Is  rated  "3." 


Matching 


a.  Directions  should  include  statements  of: 

•  the  contents  of  each  column 

•  the  basis  for  matching 

•  how  often  choices  may  be  used 

•  how  many  answers  are  possible  for  "givens" 

b.  All  choices  should  have  the  same  grammatical  structure. 

c.  Numerical  choice  should  be  in  order  of  magnitude. 

d.  The  item  should  not  require  a  great  deal  of  reading. 

e.  Choices  should  be  arranged  in  some  sensible  order,  for  example, 
alphabetically  or  logically. 

*f.  Unless  choices  can  be  used  more  than  once,  extra  entries  should  be 
included  in  the  choice  column. 
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*g.  None  of  the  entries  in  either  column  should  appear  obviously  different 
from  each  other.  One  title  should  be  appropriate  for  all  entries  in  a 
column. 

*h.  Each  alternative  solution  should  be  a  plausible  answer  for  all  or  most 
of  the  problems. 

EXAMPLE: 


ITEM:  Match  each  target  type  below  with  its  correct  symbol. 


1. 

ground  target 

a. 

CD 

2. 

friend 

b. 

0 

3. 

unknown  hostile 

c. 

This  item  is  deficient 

on  two  criteria: 

Directions  do 

not  indicate  how  often 

choices  can  be  used  (criterion  "a"),  and  extra  entries  are  not  included  in  the 
choice  column  (criterion  ”f’).  Since  the  item  is  discrepant  on  one  critical  (*) 
criterion,  rate  it  "3.** 


•  Fill-in 

a.  The  blank  should  be  at  or  near  the  end  of  the  sentence. 

*b.  One  and  only  one  word  or  phrase  should  correctly  complete  the  item, 

c.  Multiple  blanks  should  be  avoided. 

*d.  Blanks  should  require  key  words. 

*e.  Answers  should  be  specific  rather  than  general  In  nature. 

*f.  Items  should  measure  key  concepts,  not  nonessential  details. 

*g.  There  should  be  no  grammatical  or  logical  cues  to  correct  answers. 

EXAMPLE: 

ITEM:  _ Is  the  symbol  for  ’jammed  sector.* 

This  item  does  not  have  a  blank  near  the  end  of  the  sentence  (criterion  "a**). 
Since  this  deficiency  is  non-critical,  it  is  rated  **2.M 
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Short  answer 


a.  The  required  answer  should  be  short  (phrases  or  one  or  two  sentences). 

*b.  Directions  to  the  student  should  say  how  the  items  are  to  be  scored 

without  giving  cues  to  the  correct  answers. 

*c.  The  scoring  key  should  identify  allowable  synonyms  or  alternatives. 

*d.  The  question  should  be  complete  with  enough  information  for  the 

learner  to  generate  an  appropriate  answer. 

*e.  Credit  should  be  given  for  each  correct  part  of  an  answer. 

*f.  Questions  should  be  clear  so  that  all  learners  will  have  the  same 

interpretation. 

*g.  A  model  answer  should  be  written  for  each  question  as  an  aid  to 
scoring,  or  a  checklist  of  attributes  of  a  good  answer  should  be 
provided  for  use  in  scoring. 

h.  Items  should  be  assigned  points  according  to  their  value. 

i.  Point  values  should  be  made  known  before  the  test. 

J.  Items  should  be  scored  anonymously  (scorer  should  grade  students' 

papers  without  knowledge  of  their  identity). 

*k.  Content  should  be  scored,  not  grammar,  punctuation,  spelling,  or 
penmanship  (unless  specified  in  the  objective). 

1.  One  question  should  be  scored  across  all  students  before  preceeding  to 
the  next  one. 

EXAMPLE: 


ITEM: 


KEY: 


Give  the  rule  for  computing  the  mean  of  a  set  of  scores  as  a 
formula.  No  credit  for  partially  correct  answers. 


or  the  mean  = 


sum  of  the  scores 
number  of  scores 


This  item  meets  all  of  the  above  criteria.  It  should  be  rated  "1." 


Listing 

♦a.  The  directions  should  specify  the  number  of  things  to  be  listed,  (if 
appropriate  for  the  objective,  and  if  the  number  of  things  is  not  a 
clue.) 

*b.  The  directions  stauld  indicate  how  the  item  will  be  scored  without 
giving  clues  to  the  correct  answer. 
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*c.  If  order  is  important,  the  scoring  key  should  treat  sequence  separately. 

*d.  The  scoring  key  should  identify  allowable  synonyms  or  alternatives. 

*e.  Spelling  should  not  be  scored  unless  required  by  the  objective. 

f.  The  scoring  key  should  identify  relative  weights  of  different  items  on 
the  list,  if  appropriate. 


EXAMPLE: 

ITEM: 

List 

the  squad  functions  concerned  with  radar,  in  order. 

KEY: 

i. 

Set  radar  mode  switch 

2. 

Select  channel 

3. 

Select  search  sector 

4. 

Select  display  parameters 

5. 

Observe  BIT  results 

6. 

Administer  organization  maintenance 

This  item  is  deficient  on  several  critical  criteria.  The  directions  do  not 
specify  the  number  of  squad  functions  to  be  listed  (criterion  "a");  the 
directions  indicate  that  order  is  important,  but  the  key  does  not  treat 
sequence  separately  from  completeness  (criterion  "b");  the  scoring  key  does 
not  identify  allowable  synonyms  or  alternatives  O’d")*  Clearly,  this  item  calls 
for  a  "3*1  rating. 


Performance 

NOTE:  A  performance  test  question  is  the  request  that  a  task  or  parts 

of  a  task  be  performed.  It  is  not  an  individual  point  on  a 
checklist. 


*a.  The  test  should  score  all  aspects  of  task  performance  and  qualites  of 
the  resulting  task  product  that  are  specified  in  the  objective  and  no 
others. 

*b.  The  directions  should  clearly  explain  what  the  student  is  to  do  and  how 
the  item  will  be  scored  (and/or  the  test  administrator  should  clearly 
communicate  that  to  the  student). 

Any  checklists  employed  should  follow  the  rules  in  c.  &  d.: 

c.  Each  step  should  be  self-explanatory  to  anyone  who  would  perform  the 
task. 

d.  Each  step  should  be  stated  as  a  single  action;  important  or  complex 
decisions  should  be  presented  as  separate  steps. 

*e.  The  scoring  key  should  specify  a!!  criteria  the  performance  must  meet, 
such  as  completeness,  accuracy,  quatity,  time  limit,  rate,  etc. 
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1.  If  steps  In  the  performance  are  scored,  a  checklist  should  be  provided, 

*g.  Any  rating  scales  used  should  be  specific  enough  so  that  all  raters  will 

interpret  them  in  much  the  same  manner. 

*h.  All  raters  should  agree  on  judgments  of  task  performance. 

If  more  then  one  scorer  rates  the  same  students,  multiple  ratings  can  be 
compared. 

Interviews  with  scorers  on  question  irh.M  may  also  be  an  appropriate  data 
source. 


PROCESS, 

PERFORMANCE  ONLY 

IS.  When  steps  are  scored,  does  the  instructor  use  a  checklist? 

RATINGS  If  a  checklist  is  filled  in  completely,  rate  1. 

If  a  checklist  is  used  as  a  reference  or  filled  in  partially,  rate  2, 
It  a  checklist  is  not  used,  rate  3. 


PRODUCT  ONLY 
TRUE-FALSE,  MULTIPLE 
CHOICE,  OR  MATCHING 
ONLY 

19.  On  selected  response  tests  (l.e.,  true-false,  multiple  choice,  or  matching)  is 
each  correct  answer  position  (e.g.,  a,  b,  c,  or  d)  used  about  the  same  number 
of  times? 


If  one  or  more  correct  answer  positions  are  used  more  than  others,  It  can 
give  students  clues  which  make  the  test  easier  than  it  ought  to  be. 

RATING:  If  correct  answer  positions  are  used  about  the  same  number  of 

times,  rate  1. 


If  it  would  be  obvious  to  the  test  taker  that  some  correct 
answer  positions  are  used  more  than  others,  rate  3. 


PRODUCT  ONLY 
TRUE-FALSE,  MULTIPLE 
CHOICE,  OR  MATCHING 
ONLY 

20.  On  selected  response  tests  (i.e.,  true-false,  multiple  choice,  matching)  are 

specific  patterns  of  correct  answer  positions  repeated  across  test  items  or  are 
single  positions  repeated  in  blocks? 

As  is  true  when  some  correct  answer  positions  are  used  more  than  others, 
using  patterns  (e.g.,  c,  d,  a,  b,  c,  d,  a,  b);  or  repeating  single  positions  in 
blocks  (e.g.,  c,  c,  c,  c,  d,  d,  d,  d);  can  give  students  clues  and  make  the  test 
easier  than  it  should  be. 

RATING:  If  there  are  no  answer  patterns  than  can  be  seen  easily  in  the 

test,  rate  1. 

If  patterns  of  answers  can  be  seen,  rate  3. 


21.  Are  test  administration  directions  complete? 

Instructions  should  be  complete  enough  that  test  administrators  don’t  have  to 
make  up  their  own  directions,  i.e.,  so  different  testers  would  tend  to  follow 
the  same  procedures. 

RATING:  If  the  directions  are  complete,  rate  1. 

If  directions  are  provided,  but  are  incomplete  or  undear,  rate  2. 
If  there  are  no  administration  directions,  rate  3. 

PROCESS  ONLY  ~~ 


22.  Do  Instructors  follow  the  directions  when  administering  the  test?  If  there  are 
no  documented  directions  this  question  is  not  applicable. 

RATING:  If  yes,  rate  1. 

If  there  are  some  variations  from  the  test  directions,  rate  2. 

If  there  are  significant  variations  from  the  directions,  rate  h 


23.  Are  adequate  test  instructions  provided  to  the  student? 

Two  types  of  instructions  may  be  required: 

1.  Item  instructions  for  groups  of  test  items.  These  should  be  clear  and 
should  provide  enough  information  to  answer  the  items  correctly.  (Be 
sure  to  indicate  which  items  the  rating  applies  to  on  your  worksheet.) 

2.  General  instructions  should  state: 

•  the  test’s  purpose 

•  any  time  limit.  A 


•  descriptions  of  test  conditions 

•  descriptions  of  test  standards 

•  how  to  respond  to  each  item 

•  general  regulations 

There  should  be  no  questions  in  a  student’s  mind  as  how  to  take  the 
test  or  how  to  behave  during  the  test. 

RATING:  If  yes,  i.e.,  adequate  test  instructions  to  the  student  are 

provided,  rate  1. 

If  instructions  are  provided  but  are  not  completely  clear,  rate  2. 
If  no  instructions  are  provided,  rate  3. 


PERFORMANCE  ONLY 

24.  Does  the  FINAL  TEST  integrate  tasks  as  they  are  integrated  in  the  ’’real 
world?” 

If  real  world  CONDITIONS  call  for  tasks  to  be  performed  together  or  during 
the  same  period  of  time  the  test  should  simulate  those  CONDITIONS. 

RATING:  If  yes,  i.e.,  all  tasks  performed  together  on  the  job  are 

performed  together  on  the  test,  rate  1. 

If  tasks  are  only  partially  integrated,  rate  2. 

If  the  tasks  are  tested  separately,  rate  3. 


25.  Are  tasks  and  task  steps  tested  in  the  same  sequence  as  they  are  performed 
in  the  "real  world?” 

RATING:  If  yes,  i.e.,  the  sequence  is  the  same,  rate  i. 

If  some  tasks  or  steps  are  slightly  out  of  sequence,  rate  2. 

If  the  tasks  or  steps  are  in  a  very  different  sequence,  rate  3. 

PROCESS  ONLY 


26.  Is  the  test  free  of  external  cues  and  help? 

Were  there  any  artifical  cues  or  help?  Instructors,  evaluators,  observers,  and 
other  students  should  not  provide  any  assistance  that  will  not  be  present  in 
the  real  world. 

RATING:  If  yes,  i.e.,  no  cue  or  help  was  given  during  the  test  (other  than 

clarification  of  the  directions),  rate  1. 

If  some  ”hints”  are  given,  rate  2. 

If  assistance  was  given  that  would  ’’give  away”  many  correct 
answers,  rate  3.  (Additionally,  test  item  scores  should  be 
recorded  as  ”NO  GOs”  for  TEE  purposes.) 
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PRESENTATION  QUESTIONS 


27.  Are  attention-getting/motivational  techniques  employed  in  the  instruction? 

Devices,  such  as  cartoons,  interesting  subject  matter-related  stories,  and 
analogies,  can  be  used  to  arouse  interest.  There  must  also  be  appropriate 
positive  rewards  for  learning  and  negative  consequences  for  not  learning. 

These  rewards  and  consequences  may  be  a  normal  part  of  many  learning 
situations.  For  example,  the  possible  satisfaction  of  passing  the  course  or  the 
disappointment  of  failing  it  is  enough  motivation  for  many  students. 
Motivational  devices  in  the  instruction  and/or  special  rewards  and 
consequences  are  usually  required  because: 

«  the  students  often  have  little  motivation  for  learning,  such  as  many 
Category  IV  trainees. 

•  the  students  would  consider  the  task  evaluated  in  the  TEE  long  and 
boring  or  difficult  to  learn. 

RATING:  If  motivational  devices  are  included  in  the  instruction,  rate  1. 

If  motivational  devices  are  not  included,  rate  3. 


PROCESS  ONLY 


28.  Is  the  trainee  attitude  positive? 

Do  the  trainees  appear  interested  and  actively  participate  in  the  instruction 
or  do  they  do  or  say  anything  that  shows  that  they  don’t  care  about  the 
instruction  or  that  they  don’t  want  to  learn? 

RATING:  If  students  display  a  positive  attitude  toward  instruction,  rate  1. 

If  students  appear  indifferent,  rate  2. 

If  students  display  hostility  or  frustration  during  the  instruction, 
rate  3. 


PRODUCT  ONLY 


29.  Are  course  ENTRY  SKILLS  reviewed? 

Skills  and  knowledge  required  of  students  entering  the  course  should  be 
reviewed  if  they  arc  complex  or  likely  to  have  been  forgotten.  (This  TEE 
question  is  not  applicable  if  a  review  of  entry  skills  is  not  needed.) 

RATING:  If  a  review  including  PRACTICE  is  given,  rate  1. 

If  a  review  is  given  without  student  PRACTICE,  rate  2. 

If  no  review  is  given,  rate  3. 
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30.  Is  mastery  of  prerequisite  skills  verified  prior  to  new  instruction? 

Prerequisite  skills  are  subordinate  skills  which  must  be  understood  or  mastered 
in  order  to  learn  a  SUPERORDINATE  SKILL  taught  later  in  the  course.  They 
can  be  verified  by  giving  a  test  or  asking  a  sample  of  the  students  how  to 
perform  them. 

RATING:  If  prerequisite  skills  are  verified,  rate  1. 

If  prerequisites  are  not  verified,  rate  3. 


31.  Are  the  OBJECTIVES  presented  to  the  student? 

Does  the  lesson  call  for  a  statement  of  terminal  performance  required  of  the 
students  for  the  OBJECTIVE  and/or  is  the  statement  given  in  the  instruction? 

RATING:  If  the  OBJECTIVES  are  presented  to  the  student,  rate  1, 

If  no  OBJECTIVES  are  stated,  rate  3. 


ANALYST  ONLY 

32.  Are  the  basic  PRESENTATION  COMPONENTS  present? 

* 

RATING:  Rate  the  OBJECTIVE  in  accordance  with  the  tables  below.  The 

first  table  shows  the  required  components  for  each  task  level. 
The  second  table  gives  a  potential  rating. 


A-19 


In  the  table  below,  select  the  row  within  the  appropriate  task  level  that  matches 
the  training  events  used  for  the  objective.  The  boxes  have  the  following  meanings: 


’V  —  Component  present 

Blank  Box  —  Component  missing 


REQUIRED  PRESENTATION  COMPONENT 


In  the  table  below,  select  the  row  within  the  appropriate  task  level  that  matches 
the  training  events  used  for  the  objective*  The  boxes  have  the  following  meanings: 


—  Component  present 


Blank  Box  —  Component  missing 


(continued) 


REQUIRED  PRESENTATION  COMPONENT 


Task  Level 

State¬ 

ment 

Practice 

Remem¬ 

bering 

Feedback 

Examples 

Practice 

Using 

Feedback 

Rating 

USE-AIDED 

• 

© 

© 

1 

0 

© 

2 

• 

© 

1 

© 

3 

o 

3 
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33.  Are  STATEMENTS  complete? 


STATEMENTS  should  have  the  following  parts: 


©  For  FACTS: 

•  For  CONCEPTS: 

•  For  PROCEDURES: 

©  For  RULES: 

•  For  PRINCIPLES: 


the  complete  fact  or  set  of  facts  should  be 
presented. 

all  critical  characteristics  and  how  they  are 
combined  should  be  given. 

all  steps  should  be  given  in  the  correct  order. 

all  steps  and  branching  decisions  should  be 
given  In  the  correct  order. 

all  potential  component  CONCEPTS, 
PROCEDURES,  RULES,  and/or  PRINCIPLES 
should  be  given  (unless  REMEMBERING  them 
can  be  considered  an  ENTRY  SKILL)  along 
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with  the  PRINCIPLE  itself  (a  general  rule 
giving  typically  incomplete  guidance  on  how 
to  construct  a  HIGHER  ORDER  RULE  from 
the  component  parts). 

RATING:  If  all  parts  of  the  STATEMENT  are  present,  rate  i. 

If  few  parts  are  missing,  rate  2. 

If  many  parts  are  missing,  rate  3. 

ANALYST  ONLY 

34.  Are  STATEMENTS  for  CONCEPTS,  PROCEDURES,  or  RULES  adequate? 

•  For  CONCEPTS:  a  decision  rule  or  search  strategy  for 

classifying  examples  and  non-examples  should 
be  given. 

•  For  PROCEDURES:  each  step  should  have  only  one  action. 

•  For  RULES:  when  a  formula  is  used,  symbols  must  be 

defined.  i 

RATING:  If  the  STATEMENT  is  adequate,  rate  1. 

If  one  of  the  above  rules  Is  violated,  rate  3. 

35.  Does  STATEMENT  HELP  provide  sufficient  explanation? 

NOTE:  The  purpose  of  STATEMENT  HELP  as  defined  here  is  to  help  in 

understanding  the  statement  and  in  remembering  it. 

STATEMENT  HELP  may  consist  of: 

•  Explaining  how  the  statement  relates  to  something  the  student 
already  knows,  why  it  is  important,  how  it  fits  into  the  course 

•  More  explanation  about  what  the  statement  means 

•  Representing  the  statement  with  pictures,  symbo  s,  or  flowcharts 

RATING:  If  STATEMENT  HELP  gives  sufficient  explanation,  rate  i. 

If  STATEMENT  help  gives  insufficient  explanation  (e.g.,  merely 
restates  the  STATEMENT  in  the  same  words),  rate  2. 

If  STATEMENT  HELP  is  confusing,  rate  3. 
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Questions  36  through  43  apply  to  EXAMPLES.  If  the  course  consists  only  of 
OBJECTIVES  at  the  REMEMBER  TASK  LEVEL  or  of  the  FACT  CONTENT  TYPE, 
these  questions  will  not  apply. 

36.  Does  training  include  instruction  on  the  use  of  required  job  performance  aids? 

3ob  performance  aids  should  be  adequately  introduced  and  explained  so  that 
students  will  know  how  to  use  them.  They  should  be  clearly  shown  and 
referenced  in  examples  and  demonstrations. 

RATING:  If  the  job  aid  is  explained,  rate  1. 

If  the  job  Is  not  explained,  e.g.,  merely  mentioned,  rate  3. 


ANALYST  ONLY 

37.  Are  EXAMPLES  and  NON-EXAMPLES  adequate? 

•  For  CONCEPTS:  EXAMPLES  should  show  all  CRITICAL 

CHARACTERISTICS;  for  NON-EXAMPLES  it 
should  be  clear  that  one  or  more  of  the 
CRITICAL  CHARACTERISTICS  is  missing. 

•  For  PROCEDURES,  EXAMPLES  should  show  an  application  (use) 

RULE,  or  of  the  PROCEDURE,  RULE,  or 

PRINCIPLES:  PRINCIPLE. 

RATING:  If  the  EXAMPLE  or  NON-EXAMPLE  follows  thq  above  criteria, 

rate  1. 

If  the  above  criteria  are  not  followed,  rate  3. 

ANALYST  ONLY 

38.  Is  EXAMPLE  HELP  adequate? 

EXAMPLE  HELP  Is  additional  information  to  help  the  student  understand. 
Some  types  of  HELP  for  different  CONTENT  TYPES  are  given  below: 

•  HELP  for  CONCEPTS:  Highlight  CRITICAL  CHARACTERISTICS. 

Explain  the  reasons  for  classification. 
Illustrate  the  use  of  the  decision  rule  or 
search  strategy. 

t  HELP  for  Explain  the  steps.  Give  more  Information 

PROCEDURES  or  on  how  to  perform  each  step.  Give  a 

RULES:  demonstration  of  each  step.  Give  flowcharts, 

tables,  etc. 

•  HELP  for  Highlight  important  causes,  effects, 

PRINCIPLES:  relationships.  Give  additional  information. 

RATING:  EXAMPLES  which  are  more  lengthy  or  complicated  may  require 

extensive'EXAMPLE  HELP  in  order  to  receive  a  rating  of  1. 
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If  EXAMPLE  HELP  provides  sufficient  explanation,  rate  1. 
If  EXAMPLE  HELP  is  confusing,  rate  3. 
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39*  Are  EXAMPLES  sequenced  from  easy  to  hard?  (CONCEPTS  only) 

Difficulty  in  learning  CONCEPTS  can  result  either  from  a  large  number  of 
CRITICAL  CHARACTERISTICS  or  from  CRITICAL  CHARACTERISTICS  which 
are  hard  to  identify  when  they  appear  in  an  EXAMPLE.  Sequencing  of 
EXAMPLES  from  easy  to  hard  can  allow  the  student  to  learn  gradually  in 
instances  where  he  may  not  be  able  to  understand  EXAMPLES  as  they  would 
be  encountered  on  the  job. 

RATING:  If  EXAMPLES  are  sequenced  from  easy  to  hard,  rate  1. 

If  EXAMPLES  are  not  sequenced,  rate  3. 
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40.  Are  there  enough  EXAMPLES? 

There  should  be  enough  EXAMPLES  to  cover  the  content  area  thoroughly. 

•  For  CONCEPTS: 

•  For  PROCEDURES 

or  RULES: 

•  For  PRINCIPLES: 

RATING:  If  most  or  all  of  the  required  EXAMPLES  are  given,  rate  1. 

If  several  of  the  required  EXAMPLES  are  not  given,  rate  3. 
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At  least  one  EXAMPLE  for  each  CRITICAL 
CHARACTERISTIC. 

One  EXAMPLE  for  each  important  step; 
EXAMPLES  for  all  possible  branches  or 
decisions. 

At  least  one  EXAMPLE  for  each  major  type 
of  HIGHER  ORDER  RULE  that  should  be 
discovered. 


41.  Are  NON-EXAMPLES  included?  (CONCEPTS  only) 


For  CONCEPTS,  NON-EXAMPLES  should  be  included  for  common  NON- 
CRITICAL  CHARACTERISTICS. 

RATING:  If  NON-EXAMPLES  are  Included,  rate  1. 


If  NON-EXAMPLES  are  not  included,  rate  3. 
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42.  Do  DEMONSTRATIONS  show  how  to  correct/avoid  common  errors? 


If  there  are  errors  that  are  typically  made  on  the  job,  methods  for  avoiding 
or  correcting  them  should  be  included. 

Consult  a  subject  matter  expert  if  necessary  to  determine  the  likelihood  of 
common  errors. 

RATING:  If  explanations  are  given  on  how  common  errors  can  be  corrected 

or  avoided,  rate  1. 

If  not,  rate  3. 
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43.  Are  steps  in  a  DEMONSTRATION  the  appropriate  size? 

The  number  of  steps  or  concepts  that  are  demonstrated  to  trainees  at  one 
time  shoulld  not  be  too  small  or  large. 


IF  TASK  INVOLVES: 

Fine  judgments 
Motor  Skills 
Difficult  concepts 
Unlearning  similar  concepts 

Straightforward  sequence 
Simple  concepts 


IF  TRAINEES  ARE: 

Very  new  to  the  task 
Low  academic  background 

Considerable  experience  in 
the  task 

High  academic  background 


STEP-SIZE  SHOULD  BE: 

Relatively  SMALL 

Relatively  LARGE 

STEP-SIZE  SHOULD  BE: 
Relatively  SMALL 

Relatively  LARGE 


RATING:  If  step-size  is  appropriate,  rate  1. 

If  step-size  is  too  small,  rate  2. 

If  step-size  is  too  large,  rate  3. 

NOTE:  Sometimes  a  step  in  a  demonstration  will  cover  too  much  ground 

and  the  soldiers  will  not  be  able  to  follow  what  the  instructor  is 
doing.  When  the  instructor  leaves  the  soldier  behind  by  taking 
large  steps  during  the  demonstration,  soldiers  may  indicate  that 
they  do  not  understand  what  the 


instructor  did,  or  ask  a  question  that  causes  the  instructor  to  backtrack  and 
provide  additional  information.  If  the  added  information  appears  to  increase 
soldier  understanding,  then  the  step  size  used  in  the  demonstration  was 
probably  too  large.  If  so,  rate  3. 


44.  Are  tasks  and  task  steps  DEMONSTRATED  in  the  same  sequence  as  they  are 
performed  in  the  real  world? 

RATING:  If  the  order  is  the  same,  rate  1. 

If  the  DEMONSTRATION  is  slightly  out  of  sequence,  rate  2. 

If  there  is  a  very  different  sequence,  rate  3. 


Questions  45  through  68  deal  with  PRACTICE. 

Questions  45  and  46  apply  only  to  PRACTICE  REMEMBERING  and  are  therefore  not 
appropriate  for  USE-AIDED  OBJECTIVES. 

Questions  47  through  52  apply  only  to  PRACTICE  USING  and  are  not  appropriate  for 
REMEMBER  OBJECTIVES. 

Questions  51  and  52  are  appropriate  only  to  USE-AIDED  OBJECTIVES. 

Questions  53  through  64  deal  with  PRACTICE  and  FEEDBACK  In  general,  but  may 
be  appropriate  to  any  TASK  LEVEL  or  CONTENT  TYPE. 

Questions  65  through  68  deal  with  TEAM  PRACTICE  and  are  appropriate  only  when 
members  of  a  team  PRACTICE  their  overall  TEAM  FUNCTION  collectively. 


45.  Arc  memory  aids  used?  (PRACTICE  REMEMBERING  only) 

Memory  aids  may  take  several  forms: 
a  Flowcharts 

•  Visuals  such  as  graphics,  patterns,  and  illustrations 

a  Verbal  devices  such  as  rhymes,  stories,  chains  of  related  words, 
and  keywords 

RATING:  If  a  memory  aid  is  used,  rate  i. 

If  a  memory  aid  is  not  used,  rate  3. 


46.  Does  each  PRACTICE  REMEMBERING  item  have  the  same  content  and 
format  as  the  test  item? 

Here,  PRACTICE  and  test  items  should  be  identical  so  that  differences  in 
content,  wording,  illustrations,  etc.  do  not  become  a  source  of  confusion  to 
the  student. 
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RATING:  If  the  PRACTICE  REMEMBERING  item  is  the  same  as  the  test 

item,  rate  1, 

If  the  items  are  the  same  in  content  but  differ  in  any  other 
way,  rate  2. 

If  the  items  differ  in  content,  rate  3. 


47.  Are  PRACTICE  USING  Items  sequenced  from  easy  to  hard? 

Tasks  are  often  difficult  to  learn  when  their  full  CONDITIONS  and 
STANDARDS  are  imposed  on  early  PRACTICE  items.  In  order  to  teach 
someone  how  to  perform  such  Tasks,  it  is  often  necessary  to  make  it  easier 
to  do  at  the  beginning.  To  do  this,  the  CONDITIONS  or  STANDARDS  can  be 
relaxed.  The  ACTION,  however,  should  never  be  changed. 

For  example,  the  Task  "field-strip  an  M-16  rifle  in  total  darkness"  is  taught 
initially  in  the  light. 

RATING:  If  PRACTICE  items  are  sequenced  from  easy  to  hard,  rate  1. 

If  sequencing  is  not  done,  rate  3. 


ANALYST  ONLY 

48.  Do  PRACTICE  USING  items  provide  opportunities  for  COMMON  ERRORS  to 
be  made? 

The  test  items  should  test  COMMON  ERRORS  (see  Question  10).  The 
PRACTICE  Items  should  allow  for  those  same  errors  to  be  made. 

RATING:  If  PRACTICE  USING  items  allow  COMMON  ERRORS  to  be 

made,  rate  1. 

If  the  items  provide  no  opportunity  for  COMMON  ERRORS  to  be 
made,  rate  3. 
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49.  Are  PRACTICE  items  different  from  EXAMPLES?  (USE-CONCEPT,  USE- 
RULE,  or  USE-PRINCIPLE  only) 

PRACTICE  with  situations  that  have  been  seen  before  in  EXAMPLES  does  not 
allow  the  correct  type  of  learning  to  occur  for  USE  OBJECTIVES. 

RATING:  If  PRACTICE  USING  items  have  never  been  seen  before,  rate  l. 

If  PRACTICE  items  have  been  seen  before,  rate  3. 
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50. 


Does  PRACTICE  USING  integrate  tasks  as  they  are  integrated  in  the  "real 
world?” 


Make  one  rating  per  group  of  terminal  learning  objectives  (TLOs)  in  terms  of 
how  well  the  PRACTICE  situation  matches  that  of  the  job.  You  may  need  to 
consult  a  subject  matter  expert.  Be  sure  to  record  the  task/TLO  #s  on  your 
worksheet. 

RATING:  If  PRACTICE  integrates  tasks  as  they  are  integrated  in  the  "real 

world,”  rate  1. 

If  tasks  are  only  partially  integrated,  rate  2. 

If  tasks  are  practiced  separately,  rate  3. 

51.  Are  job  performance  aids  (3PAs)  usable? 

The  following  points  should  be  true  of  all  JPAs: 

•  Each  step  is  self-explanatory  to  anyone  who  would  perform  the 
task. 

•  Each  step  picks  up  where  the  previous  step  left  off. 

a  Illustrations  that  plainly  identify  components  to  be  manipulated 
or  identified  are  included  where  applicable. 

RATING?  If  the  job  aid  is  easy  to  use,  rate  U 

If  the  job  aid  Is  hard  to  use,  rate  2. 

_ _ If  the  job  aid  is  unusable,  rate  3. 

PROCESS  ONLY 


32.  Do  all  students  use  the  job  performs  >ce  aid  (3PA)? 

RATING:  If  all  students  use  the  3PA,  rate  1. 

If  up  to  20%  of  the  students  do  not  use  the  3PA,  rate  2. 

If  mere  than  20%  of  the  students  do  not  use  the  !fPA,  rate  3. 


53.  Docs  the  TASK  LEVEL  of  the  PRACTICE  item  match  that  of  the  test 

ltem(s)?  Compare  the  ACHONS  of  the  PRACTICE  item  and  test  item(s)  that 
go  with  it. 
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RATING 


Test  Item  TASK  LEVEL 


REMEMBER 

USE-UNAIDED 

USE-AIDED 

PRACTICE 

ITEM 

TASK 

LEVEL 

REMEMBER 

1 

2 

2 

USE-UNAIDED 

2 

1 

2 

USE-AIDED 

3 

3 

1 
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54.  Does  the  CONTENT  TYPE  of  the  PRACTICE  item  match  that  of  the  test 
item(s)? 

CONTENT  TYPE  refers  to  whether  the  objective  deals  with  a  FACT, 
CONCEPT,  PROCEDURE,  RULE,  or  PRINCIPLE. 

RATING:  If  the  CONTENT  TYPE  matches  (e.g.,  CONCEPT-CONCEPT)  rate 

1. 

If  there  is  a  mismatch  (e.g.,  CONCEPT-RULE)  rate  3. 


55.  Does  the  FORMAT  of  the  PRACTICE  item  match  that  of  the  test  item?  j 

RATING:  1 


Test  Item  Format 
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56.  Do  the  CONDITIONS  of  each  FINAL  PRACTICE  item  match  those  of  the  test 
itemfe)? 

Compare  the  CONDITIONS  of  FINAL  PRACTICE  to  those  of  the  test  items 
that  test  it*  This  question  applies  only  to  the  latest  PRACTICE  given  before 
the  test* 

RATING:  If  CONDITIONS  are  the  same,  rate  I. 

If  CONDITIONS  are  slightly  different,  rate  2. 

If  CONDITIONS  are  very  different,  rate  3. 


57*  Do  the  STANDARDS  of  each  FINAL  PRACTICE  item  match  those  of  the  test 
item(s)? 

Compare  the  STANDARDS  imposed  during  FINAL  PRACTICE  to  those  used  in 
the  test  items  that  go  with  it.  This  question  applies  only  to  the  last 
PRACTICE  given  before  the  test. 

RATING:  If  STANDARDS  are  the  same,  rate  1. 

If  STANDARDS  are  slightly  different,  rate  2. 

If  STANDARDS  are  very  different,  rate  3. 
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5S.  Is  FINAL  PRACTICE  free  of  external  cues  or  help? 

FINAL  PRACTICE  should  not  contain  anything  that  will  not  appear  on  the 
test  or  on  the  job.  The  instructors,  evaluators,  observers,  and  other  students 
should  not  provide  assistance  that  won!t  be  present  in  the  "real  world."  Extra 
hints  given  during  practice  will  cause  more  students  to  fail  the  test  (once 
they  don’t  have  the  assistance  to  rely  on  when  taking  the  test).  This 
question  applies  only  to  the  last  PRACTICE  given  before  the  test. 

RATING:  If  yes,  i.e.,  no  assistance  is  given  during  PRACTICE,  rate  1. 

If  some  hints  are  given  that  wouldn’t  be  available  on  the  job, 
rate  2. 

If  students  are  given  excessive  cues,  prompts,  or  assistance  such 
that  many  answers  are  given  away,  rate  3. 


ANALYST  ONLY 

59.  Are  there  PRACTICE  items  for  each  TLO  or  all  of  its  critical  parts/LOs? 

NOTE:  As  defined  in  Section  4  of  the  Guidelines  for  evaluating 

objectives  in  Task  A4,  cr'tical  parts/LOs  have  the  following 
characteristics: 
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•  They  are  some  of  the  most  measurable  and  observable 
elements  of  the  task. 

•  They  have  serious  consequences  of  inadequate  performance 
in  terms  of  their  impact  on  task  performance. 

•  They  are  common  sources  of  failure. 

•  They  are  actions  which  the  soldiers  of  the  target 
population  do  not  already  know  how  to  do  and  must 
therefore  be  taught. 

RATING:  If  there  are  PRACTICE  items  for  the  TLO  or  all  of  its  critical  * 

parts./LOs,  rate  1. 

If  there  are  no  PRACTICE  items  for  the  TLO  and  only  some 
critical  parts/LOs,  rate  2. 

If  there  are  no  PRACTICE  items  for  the  TLO  and  for  most  of 
the  parts/LOs,  rate  3. 


ANALYST  ONLY 

60.  Is  there  a  PRACTICE  item  for  each  critical  part  of  each  LO? 

The  critical  parts  of  LOs  are  defined  below. 

For  the  REMEMBER  TASK  LEVEL  each  piece  of  information  is  critical.  Each 
of  the  following  should  be  tested: 

•  Facts 

•  Concept  characteristics 

•  Procedural  steps 

•  Steps  and  conditions  for  branching  in  a  rule 

•  Guidelines  in  a  principle 

For  the  USE-AIDED  and  USE-UNAIDED  TASK  LEVELS,  the  parts  of  the 
OBJECTIVE’S  ACTION  are  critical: 

•  For  CONCEPTS,  are  all  CRITICAL  CHARACTERISTICS 

practiced?  Can  you  be  sure  the  student  knows  each  one  and  can 
you  identify  which  ones  he  does  not  know? 

•  For  PROCEDURES,  is  each  of  the  steps  performed? 

•  For  RULES  and  PRINCIPLES,  is  each  of  the  steps  performed? 

Are  there  items  for  all  possible  branches  or  decisions,  or  at  least 
those  that  would  commonly  occur  on  the  job? 
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RATING:  If  there  is  PRACTICE  for  all  parts,  rate  1. 

If  there  is  PRACTICE  for  many,  but  not  all  parts,  rate  2. 

If  there  is  PRACTICE  for  only  a  few  parts  or  for  no  parts,  rate 
3. 
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61.  Do  all  students  PRACTICE? 

RATING:  If  yes,  rate  1. 

If  up  to  20%  of  the  students  do  not  PRACTICE,  rate  2. 

.  If  more  than  20%  of  the  students  do  not  PRACTICE,  rate  3. 


PROCESS  ONLY 

62.  Do  ail  students  meet  the  required  STANDARDS  in  FINAL  PRACTICE? 

This  question  applies  only  to  the  last  practice  given  before  the  test. 

RATING:  If  yes,  rate  1. 

If  up  to  20%  of  the  students  do  not  meet  the  required 
STANDARDS,  rate  2. 

If  more  than  20%  of  the  students  do  not  meet  the  STANDARDS, 
rate  3. 


63.  Is  FEEDBACK  provided  for  PRACTICE? 

For  both  PRACTICE  REMEMBERING  and  PRACTICE  USING,  students  should 
be  told  whether  they  are  right  or  wrong,  and  if  wrong,  what  the  correct 
answer  is  and  why. 

For  step-by-step  practice,  FEEDBACK  should  be  given  for  each  step. 

RATING:  If  FEEDBACK  HELP  (which  explains  answers)  is  given  for  wrong 

answers  or  correct  answer  only  feedback  is  given  for  correct 
responses,  rate  1. 

If  students  are  given  only  the  correct  answer  after  incorrect 
responses,  rate  2. 

If  no  FEEDBACK  is  provided,  rate  3. 
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6*.  Is  FEEDBACK  HELP  adequate? 

Students  sometimes  make  errors  because  they  did  not  learn  from  the  original 
presentation.  Therefore,  when  possible,  FEEDBACK  should  include  additional 
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information  or  a  different  version  of  the  information  than  was  originally 
presented. 

FEEDBACK  HELP  should  include  additional  information  to  help  the  student 
understand.  Some  types  of  HELP  for  different  CONTENT  TYPES  are  given 
belows 


HELP  for  CONCEPTS: 


HELP  FOR  PROCEDURES 
or  RULES: 


HELP  FOR  PRINCIPLES: 


Hight light  CRITICAL  CHARACTERISTICS. 
Explain  the  reasons  for  classification. 
Illustrate  the  use  of  the  decision  rule  or 
search  strategy. 


Explain  the  steps. 

Give  more  information  on  how  to  perform 
each  step. 

Give  a  demonstration  of  each  step. 

Give  flowcharts,  tables,  etc. 

Highlight  important  causes,  effects, 
relationships. 

Give  additional  information  about  how  the 
principle  applies,  or  why  it  doesn’t. 


RATING:  If  FEEDBACK  HELP  gives  enough  additional  explanation,  rate  1. 


If  it  gives  insufficient  explanation,  e.g.,  if  it  consists  of  a 
repetition  of  the  original  presentation,  rate  2. 

If  FEEDBACK  HELP  is  confusing,  rate  3. 


63.  Is  TEAM  PRACTICE  provided? 

PRACTICE  is  the  fundamental  component  of  team  training.  TEAM 
PRACTICE  allows  for  all  of  the  members  of  a  crew  or  team  to  get  together 
and  PRACTICE  their  TEAM  FUNCTION. 

RATING:  If  TEAM  PRACTICE  is  provided,  rate  1. 

If  there  is  none,  rate  3. 


66.  Are  TEAM  PRACTICE  CONDITIONS  the  same  as  (or  as  close  as  possible  to) 
those  of  the  real  task? 

When  the  members  of  a  crew  or  team  come  together  to  practice  their  TEAM 
FUNCTION  collectively,  it  should  be  done  under  CONDITIONS  as  close  to  the 
real  task  as  possible. 

If  PRACTICE  CONDITIONS  are  not  close  to  those  in  the  real  task,  important 
collective  skills  such  as  communication  and  coordination  among  team  members 
may  not  be  practiced  adequately. 
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RATING: 


II  TEAM  PRACTICE  CONDITIONS  are  as  close  as  possible  to 
those  of  the  real  task,  rate  1. 

If  CONDITIONS  are  slightly  different,  rate  2. 

If  CONDITIONS  are  very  different,  rate  3. 

NOTE:  Be  sure  to  record  the  correct  TEAM  FUNCTION  number  in  the 

Task/TLO  //  column  on  your  product  evaluation  worksheet. 


67.  Is  TEAM  PRACTICE  FEEDBACK  provided? 

Teams  should  at  least  be  told  whether  their  PRACTICE  mission  was  successful 
or  not,  and  if  not,  what  they  did  wrong. 

RATING:  If  FEEDBACK  HELP  is  provided,  rate  1. 

If  team  members  are  told  "how  they  did",  but  are  not  told  what 
they  did  wrong,  rate  2. 

If  no  FEEDBACK  is  given,  rate  3. 

NOTE:  Be  sure  to  record  the  correct  TEAM  FUNCTION  number  in  the 

Task/TLO  #  column  on  your  product  evaluation  worksheet. 
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68.  Is  FEEDBACK  HELP  for  TEAM  PRACTICE  adequate? 

In  order  for  team  members  to  learn  from  their  mistakes,  they  must  be  told 
how  to  correct  them.  Some  types  of  HELP  for  different  CONTENT  TYPES 
are  given  below: 

HELP  FOR  PROCEDURES 

or  RULES:  Explain  the  steps. 

Give  more  information  on  how  to  perform 
each  step. 

Give  a  demonstration  of  each  step. 

Give  flowcharts,  tables,  etc. 

HELP  FOR  PRINCIPLES:  Highlight  important  causes,  effects, 

relationships. 

Give  additional  information  about  how  the 
principle  applies,  or  why  it  doesn't. 

RATING:  If  FEEDBACK  HELP  gives  enough  additional  explanation,  rate  1. 

If  it  gives  insufficient  explanation,  rate  2. 

If  FEEDBACK  HELP  is  confusing,  rate  3. 
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Questions  69  through  72  deal  with  the  clarity  and  comprehensibility  of  the 
instruction*  Unless  objectives  vary  a  great  deal  on  these  points,  ask  these  questions 
for  whole  LESSONS  at  a  time. 
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69.  Are  all  PRESENTATION  COMPONENTS  separated  and  identified? 

Each  COMPONENT  should  be  separated  from  the  rest  of  the  instruction,  for 
example: 

•  Set  off  with  a  box. 

•  Different  color,  type  face,  or  underlining. 

•  Placed  on  a  separate  page  or  in  a  special  place  on  a  page. 

•  For  audio  or  film,  pause  before  introducing  the  COMPONENT. 

Each  COMPONENT  should  be  identified: 

STATEMENTS,  EXAMPLES,  and  PRACTICE  must  be  identified  so  the  student 
knows  what  they  are,  and  can  locate  them.  Labels  can  be  used  to  identify 
different  COMPONENTS: 

Definition  of  .  .  .  Example  Practice 

Procedure  .  .  .  Demonstration  Test  Yourself 

Key  Point 

These  points  are  likely  to  affect  student  performance  only  if  there  is  a 
limited  amount  of  time  to  adequately  complete  the  Instruction  or  if  the 
students  are  not  good  readers  (when  print  is  involved). 

RATING:  If  each  PRESENTATION  COMPONENT  is  separated  and 

identified,  rate  1. 

If  some  of  the  COMPONENTS  are  not  clearly  identified,  rate  2. 

If  most  or  all  of  the  COMPONENTS  are  not  clearly  identified, 
rate  3. 


70.  Is  the  technical  quality  of  written  or  spoken  material  adequate? 

Whether  text  appears  in  print,  in  some  other  vUual  medium,  as  narration,  or 
as  spoken  in  a  lecture  or  demonstration,  it  should  meet  the  following 
standards: 

a.  Each  COMPONENT  should  be  directed  at  the  primary  audience. 

b.  The  instruction  should  be  performance-oriented  rather  than  topic- 
oriented.  It  should  tell  Mwhat  to  do*'  or  "how  to  do  it”,  rather  than 
telling  about  a  task. 

c.  Main  points  should  stand  out.  They  should  not  get  lost  in  detail,  so 
that  secondary  points  appear  equal  to  main  points.  COMPONENTS 


should  not  be  too  wordy,  either  because  there  is  too  much  detail  (more 
than  the  essentials  needed  to  perform  the  job),  or  because  the  material 
is  redundant  (with  information  being  needlessly  repeated). 

d.  Information  should  be  presented  in  an  orderly  manner  so  that  it  is  not 
confusing.  Check  for: 

•  Redundancy  between  paragraphs  or  ideas 

•  Missing  information 

•  Scattered  information  —  information  about  the  same  main  point 
not  in  one  place 

•  Remote  references  —  references  to  text  or  illustrations  in  other 
places 

e.  Visual  aids  should  communicate  something.  They  must  function  in  one 
or  more  of  the  following  ways; 

•  Substitute  for  text  or  narration 

•  Lighten  the  burden  of  detail  in  text  or  narration 

•  Reinforce  or  summarize  Information  in  text  or  narration 

f.  Is  there  a  run-together  format?  A  solid  mass  of  print  should  be 
"chunked”  visually  so  that  separate  points  can  be  easily  identified. 

Items  in  audio  visual  presentations  or  lectures  should  also  be 
distinguishable. 

g.  Perspective  —  In  visual  media  do  visuals  provide  the  viewer  with  an 
adequate  frame  o"  reference  with  which  to  judge  size  and  distance9 

RATING:  Considering  the  criteria  listed  above,  if  most  or  all  of  the 

applicable  criteria  are  met,  rate  1. 

If  several  applicable  criteria  are  not  met,  rate  2. 

If  few  applicable  criteria  are  met,  rate  3. 

NOTE:  Be  sure  to  note  what  the  specific  problems  are  in  the 

Description  column  of  your  worksheet. 

NOTE:  You  should  make  this  rating  for  an  entire  LESSON  unless 

material  for  certain  OBJECTIVES  merits  different  ratings. 


71.  Is  the  wording  of  written  or  spoken  material  easy  for  the  student  to 
understand? 

Verbal  instruction  should  not  contain  too  many  hard  words,  or  many  long 
sentences.  Technical  words  related  to  the  subject  matter  are  acceptable  as 
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long  as  they  are  explained  or  the  students  have  had  previous  training,  so  that 
they  already  know  what  they  mean. 

RATING:  If  the  wording  is  easy  to  understand  with  few  hard  words  or  long 

sentences,  rate  1. 

If  there  are  some  hard  words  and  long  sentences,  rate  2. 

If  there  are  many  hard  words  and  long  sentences,  rate  3. 


72.  Is  the  instructor’s  presentation  or  the  narration  easy  to  listen  to? 

For  spoken  material  to  be  easy  to  listen  to,  its  pace  and  pitch  should  be 
varied.  It  should  not  go  on  constantly  at  the  same  rate  or  at  the  same  pitch 
so  that  is  becomes  monotonous.  Also,  continuous  repetition  of  short  patterns 
of  pace  and/or  pitch  which  give  speech  a  "sing-song”  effect  should  be  avoided. 


In  addition,  speech  should  be  understandable.  It  should  not  be  too  soft; 
speakers  should  not  have  speech  impediments,  or  heavy  accents,  nor  should 
they  mumble. 

RATING:  If  the  instructor’s  speech  or  the  narration  is  easy' to  listen  to, 

rate  1. 

If  speech  is  dull  and  monotonous,  rate  2. 

If  speech  is  hard  to  listen  to,  rate  3. 


73.  Is  the  instructor’s  presentation,  the  narration,  or  text  material  supported  by 
visuals? 

In  order  to  maintain  attention  and  interest  and  to  aid  students  in 
understanding  the  material,  verbal  presentations  should  be  supported  by 
visuals.  Visuals  may  include  real  equipment. 

RATING:  If  the  presentation  is  completely  supported  by  visuals,  rate  i. 

If  it  is  partially  supported  by  visuals,  rate  2. 

If  the  presentation  is  not  supported  by  visuals  at  all,  rate  3. 


74.  Are  visuals  easily  understood? 

RATING:  If  visuals  are  easily  understood,  rate  1. 

If  visuals  arc  understandable  with  effort,  rate  2. 
If  visuals  are  very  hard  to  understand,  rate  3. 


A-38 


■nr»rm  u  m  rjii  m  nfAV-t  vur  A 


ANALYST  ONLY 


75.  Are  OBJECTIVES  (TLOs  and  LOs)  within  each  LESSON  sequenced  properly? 

Are  the  OBJECTIVES  which  are  prerequisites  to  other  OBJECTIVES  taught 
first?  An  OBJECTIVE  is  a  prerequisite  if  it  must  be  mastered  in  order  for 
the  other  OBJECTIVE  to  be  learned.  Examples: 

•  Being  able  to  add  is  a  prerequisite  to  being  able  to  multiply. 

«  The  ability  to  locate  a  stationary  object  in  a  VULCAN  sight  is  a 

prerequisite  to  learning  how  to  track  an  approaching  aircraft 
smoothly. 

•  Knowing  the  location  and  function  of  the  controls  and  indicators 
on  a  tracked  vehicle  must  come  before  learning  how  to  drive 
one. 

Tasks  should  also  be  taught  in  the  order  in  which  they  normally  occur  on  the 
job. 

RATING:  Rate  each  LESSON.  If  the  TLOs  and  LOs  within  a  lesson  are 

sequenced  properly,  rate  1. 

If  the  OBJECTIVES  are  not  sequenced  properly,  rate  3. 
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76.  Are  the  LESSONS  sequenced  within  the  course? 

LESSONS  that  teach  tasks  which  are  SUBORDINATE  SKILLS  to  tasks  taught 
in  other  LESSONS  should  be  taught  first. 

Look  at  each  LESSON  starting  at  the  beginning  of  the  course  and  ask  what 
the  students  must  already  know  in  order  to  learn  the  TASKS  in  the  LESSON 
(the  LESSON  prerequisites). 

RATING:  If  the  LESSON  prerequisites  were  taught  in  previous  LESSONS  or 

are  ENTRY  SKILLS,  rate  1. 

If  not,  rate  3  and  note  what  knowledge  or  s’  Hs  should  have 
been  taught  earlier  In  the  Description  column  of  your  worksheet. 
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77.  Are  the  media  appropriate  for  the  OBJECTIVES? 

Compare  the  media  used  with  the  appropriate  media  in  the  table  below. 
Consider  groups  of  OBJECTIVES  that  fall  into  the  same  general  TASK  LEVEL 
and  CONTENT  TYPE  as  defined  in  the  table. 
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RATING: 


If  the  media  are  appropriate,  rate  1. 


If  not,  rate  3  and  write  the  key  words  which  are  underlined  in  the 
right-hand  column  below  in  the  Description  column  of  your 
worksheet. 


If: 

And  Task  Level/ 
Content  Type 

Is: 

Then 

Acceptable 

Media  Are:  a 

Problem  With 

Mismatch 

Consequences 
of  task  error 
are  serious,  b 

USE 

Large  Equipment 
Portable  Equipment 
Simulator 

(Required  for  final 
practice  in  addition 
to  other  media  below 
used  earlier  in 
training.) 

No  real  practice.  Other 
media  cannot  provide 
the  realism  needed  to 
PRACTICE  the  task 
adequately  under  these 
conditions. 

Self-Instruction 
(the  course 
manager  does 
not  give 
FEEDBACK  c 
to  students) 

REMEMBER 

Audio 

Filmstrip 

Motion  Picture 

Printed  Text 

Slide/Tape 

Training  Aid 

TV  Cassette 

Computer 

Programmed  Text 
Interactive  TV 

Training  Device 
(supplemented  with 
printed  text) 

Portable  Equipment 
(supplemented  with 
printed  text) 

Incomplete  information. 
Other  media  would  not 
typically  present  enough 
information  without  the 
aid  of  an  instructor. 

USE/MENTAL 
SKILLS  d 

Computer 

Program  md  Text 
Interactive  TV 

Training  Device 

Motion  Picture 
Slide/Tape 

TV  Cassette 

Filmstrip 

Audio 

No  FEEDBACK.  Other 
media  typically  cannot 
provide  corrective 
FEEDBACK  without 
an  instructor. 

USE/MOTOR 
SKILLS  e 

Training  Device 
(supplemented  with 
printed  text) 

Portable  Equipment 
(supplemented  with 
printed  text) 

No  hands-on  practice. 

Other  media  cannot 
provide  the  necessary 
motor  manipulations. 

If: 

And  Task  Level/ 
(Content  Type) 

Is: 

Then 

Acceptable 

Media  Are:  a 

Problem  With 
Mismatch 

Instructor-led 

instruction 

REMEMBER 

Instructor  with  any 
other  media 

— 

USE/MENTAL 
SKILLS  d 

Instructor  with  any 
other  media 

— 

USE/MOTOR 
SKILLS  e 

Instructor  with: 
Portable  Equipment, 
Large  Equipment, 
Training  Device, 
or  Simulator 

No  hands-on  practice. 
Other  media  cannot 
provide  the  necessary 
motor  manipulations 

a  Media  definitions  may  be  found  in  Appendix  E  at  the  end  of  these  guidelines. 

41  When  the  graduate  is  first  required  to  perform  the  task  on  the  job,  are  the 
consequences  of  error  serious? 

c  A  course  manager  would  do  no  more  than  grade  tests.  He  would  not  give 
FEEDBACK  on  PRACTICE.  The  media  must  do  that  in  this  setting. 

d  Mental  skills  are  CONCEPTS,  PROCEDURES,  RULES,  or  PRINCIPLES  which  are 
performed  primarily  in  the  students  mind.  A  skill  may  have  both  mental  and 
motor  skills  associated  with  it,  but  if  the  motor  skills  are  so  simple  that  anyone 
who  knew  the  mental  skills  could  perform  them  without  training,  the  overall  skill 
should  be  considered  a  mental  skill.  Sometimes  a  TLO  is  broken  down  into 
mental  and  motor  LOs. 

e  Motor  skills  are  PROCEDURES  which  require  the  smooth  timing  of  muscular 
movements.  All  motor  skills  have  a  mental  component,  but  with  motor  skills  the 
physical  activity  must  be  practiced  in  order  to  achieve  proficiency. 


NOTE:  All  of  the  media  shown  for  each  situation  are  not  necessarily  the  best 

media  in  terms  of  costs  and  practicality}  however  they  can  provide  the 
necessary  learning  characteristics  for  those  conditions. 
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78,  Can  the  media  used  provide  all  necessary  stimuli  required  by  the 
OBJECTIVES?  '  ~ 

Some  OBJECTIVES  make  specific  demands  of  the  media  used  to  teach  them. 

In  these  cases,  the  media  must  be  able  to  display  certain  types  of  stimuli  in 
order  for  students  to  be  able  to  learn  the  OBJECTIVE,  These  can  include: 

•  visuals 

•  motion 

•  color 

•  audio 

•  tactile  cues  (stimuli  that  can  be  felt  or  touched) 

Note  that  this  question  deals  with  stimuli  that  are  absolutely  required  in 
order  to  teach  or  perform  the  OBJECTIVE,  For  example,  in  an  aircraft 
recognition  task,  the  use  of  visuals  is  a  requirement.  Students  cannot  iearn 
to  recognize  different  aircraft  without  seeing  visual  examples  of  them  and 
without  practicing  recognition  using  visuals. 

In  order  to  learn  how  to  load  ammunition  into  the  Vulcan  ammunition  storage 
drum,  visual  examples  are  not  required.  Color,  motion,  and  audio  would  add 
realism  and  may  be  used,  but  they  are  not  absolutely  necessary.  However, 
PRACTICE  for  that  task  would  require  tactile  cues.  The  medium  employed 
here  would  probably  be  the  operational  equipment  or  a  training  device  and  an 
instructor. 

Consider  each  medium  used  for  each  TLO  in  the  LESSON  for  the  EXAMPLES 
and  PRACTICE  items. 

RATING:  If  the  medium  can  supply  all  of  the  required  characteristics,  rate 

1. 

If  the  medium  cannot  supply  all  of  the  required  characteristics, 
rate  3. 
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79.  Are  course  administration  directions  complete? 

Course  administrative  directions  should  be  examined  to  insure  that  they  arc 
complete. 

RATING:  If  the  directions  are  complete,  rate  1. 

If  the  directions  are  partially  incomplete,  rate  2. 

If  directions  arc  incomplete  or  nonexistent,  such  that  instructors 
arc  left  to  use  their  own  judgment,  rate  3. 
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S3.  Is  ail  of  the  content  in  the  LESSON  materials  covered? 

Compare  the  lesson  plan,  student  manual,  operator's  manual  and  other 
applicable  LESSON  materials  with  the  content  the  instructor  presents.  Be 
sure  to  note  the  nature  of  departures  from  LESSON  materials  in  the 
Description  column  of  your  worksheet.  In  self-paced  settings,  does  the  course 
manager  insure  that  all  of  the  LESSON  materials  are  covered? 

RATING:  If  all  of  the  content  found  in  the  LESSON  materials  Is  covered, 

rate  1. 

If  much  of  the  content  is  covered,  rate  2,  and  indicate  what  was 
left  out. 

If  very  little  of  the  content  in  the  LESSON  materials  is  covered, 
rate  3,  and  note  what  was  omitted. 
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S4.  Did  the  instructor  limit  his  teaching  to  the  content  in  the  LESSON  materials? 

RATING:  If  the  instructor  limited  his  presentation  to  the  LESSON 

materials,  rate  1. 

If  not,  rate  3  and  briefly  note  anything  else  he  taught. 
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S3.  h  there  enough  space  for  ail  of  the  trainees? 

•  Can  everyone  see  and  hear? 

•  Where  applicable,  does  everyone  have  an  opportunity  to  practice? 

•  Is  there  adequate  "elbow  room?" 

RATING:  If  there  is  enougn  space  for  all  the  trainees,  rate  1. 

If  it  is  a  little  crowded,  rate  2. 

If  it  is  very  cramped  or  some  students  cannot  fit  in  the  space  at 
alt,  rate  3. 
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86.  Is  the  instruction  free  of  distractions? 

Do  sound  or  visual  distractions  or  interruptions  prevent  students  from 
learning? 

Listed  below  are  some  examples  of  distractions  that  commonly 
occur  in  the  training  environment: 

•  Noisy  training  site 

•  Other  observers  present 

•  Instructor  or  soldiers  being  trained  leave  before  class  is 
complete 

•  Soldiers  do  pushups  in  class  to  stay  awake 

•  Equipment  breakdowns 

•  Delays  due  to  weather 

•  Delays  due  to  missing  equipment 

RATING:  If  the  instruction  is  free  of  distractions,  rate  I. 

If  there  are  some  distractions  which  are  merely  annoying,  rate  2. 

If  distractions  seriously  interfere  with  or  interrupt  the 
instruction,  rate  3. 

PROCESS  ONLY 

87.  Is  the  lighting  appropriate  for  the  training  situation? 

Is  there  too  little  or  too  much  light  for  reading  when  required  or  to  see 
audio-visual  displays  or  equipment? 

NOTE:  If  the  "real  work'  performance  CONDITIONS  require  low  light, 

then  advanced  PRACTICE  events  may  need  to  be  in  comparable 
lighting  conditions. 

RATING:  If  the  lighting  level  is  appropriate,  rate  1. 

If  the  lighting  is  a  little  too  dim  or  too  bright,  so  that  students 
have  trouble  reading  or  seeing  displays  and  equipment,  rate  2. 

If  the  lighting  is  so  dim  or  bright  that  students  cannot  read  or 
see  displays  and  equipment,  rate  3. 
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Is  the  temperature  appropriate  for  the  training  situation? 

Are  the  trainees  so  warm  that  they  have  trouble  concentrating  on  the 
instruction,  or  so  cold  that  they  are  more  interested  in  keeping  warm  than  in 
paying  attention  to  the  instructor? 

NOTE:  PRACTICE  may  need  to  occur  under  real  world  CONDITIONS. 

RATING:  If  the  temperature  is  appropriate,  rate  1. 

If  the  temperature  makes  students  uncomfortable,  rate  2. 

If  the  temperature  seriously  interferes  with  learning,  rate  3. 

EXAMPLE:  If  a  soldier  is  shivering  or  sweating  a  lot,  or  complains  about 
being  uncomfortable  because  of  the  weather,  rate  ”2.” 

If  weather  conditions  result  in  training  being  stopped  or 
postponed,  rate  ”3." 

PROCESS  ONLY 

Is  the  instructors  attitude  positive? 

Does  the  instructor  behave  in  a  manner  that  enhances  the  students* 
motivation  for  learning? 

•  Does  the  instructor  encourage  questions? 

•  Is  he  helpful? 

•  Respectful  of  the  trainees? 

•  Tolerant  of  failure? 

•  Available  for  assistance? 

RATING:  If  the  instructor’s  attitude  is  positive,  rate  1. 

If  not,  rate  3. 

PROCESS  ONLY 

Are  frequent  breaks  provided? 

Are  rest  periods  of  at  least  5-10  minutes  provided  every  hour,  and  are  longer 
breaks  allowed  once  the  instruction  exceeds  3  or  4  hours? 

RATING:  If  adequate  rest  periods  are  provided,  rate  1. 

If  breaks  are  provided,  but  they  are  too  short  or  infrequent,  rate 

2. 

If  breaks  are  not  provided,  rate  3. 
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91.  Is  the  speed  of  presentation  appropriate? 

The  rate  at  which  STATEMENTS,  EXAMPLES,  or  HELP  are  given  should  not 
be  so  fast  that  students  cannot  readily  understand,  or  so  slow  that  they 
become  bored. 

RATING:  If  the  speed  of  presentation  is  appropriate,  rate  1. 

If  too  slow,  rate  2. 

If  too  fast,  rate  3. 
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92.  Was  the  allotted  training  time  too  long  or  too  short? 

Long  training  sessions  in  which  the  type  of  activity  engaged  in  by  the  soldier 
do?$  not  change,  combined  with  infrequent  breaks,  can  result  in  soldiers 
becoming  bored  or  fatigued.  The  allotted  time  is  too  long  when  soldiers  meet 
all  the  training  objectives  well  before  the  class  is  supposed  to  end  (i.e.,  more 
than  one  fourth  of  the  allotted  time  remains  when  the  class  ends). 

Training  sessions  that  are  too  short  may  not  allow  enough  practice  time  or 
otherwise  reduce  the  amount  of  training  given.  Unless  there  was  sufficient 
training  time  for  all  soldiers  to  perform,  unassisted,  the  tasks  listed  in  the 
training  objectives  at  least  one  time  to  standard,  the  allotted  time  was  too 
short  for  the  training  objectives. 

If  the  training  manual  or  other  material  calls  for  too  much  or  too  little 
trianing  time,  then  this  is  a  problem  with  the  training  PRODUCT.  If  the 
instructor  chose  to  spend  too  much  or  too  little  time,  then  this  is  a  problem 
with  the  PROCESS  of  training. 

RATING:  If  the  training  time  Is  the  appropriate  length,  rate  1. 

If  it  is  too  long,  rate  2. 

_ If  It  is  too  short,  rate  3. _ 


PROCESS  ONLY 

93.  Does  equipment  used  In  the  training  function  properly? 

If  training  devices  (including  real  equipment  used  during  the  training)  do  not 
function  as  they  should,  the  learning  process  can  be  degraded.  For  example, 
in  a  task  involving  meter  calibration,  if  the  equipment  is  not  operating  as  it 
should,  students  will  not  be  able  to  see  the  task  demonstrated  properly  or 
practice  the  task  under  normal  conditions. 

RATING:  If  all  of  the  equipment  normally  used  during  the  training  events 

work  properly,  rate  1. 


If  some  of  the  equipment  malfunctions,  but  the  nature  of  tasks  demonstrated, 
practiced,  or  tested  is  not  substantially  changed,  rate  2  and  note  what  went 
wrong  in  the  Description  column  of  your  worksheet. 

If  equipment  problems  substantially  change  the  nature  of  tasks  demonstrated, 
practiced,  or  tested,  rate  3  and  note  what  went  wrong  in  the  Description 
column  of  your  worksheet. 

EXAMPLE:  If  the  instructor  announces  that  the  equipment  is  not  working 
and  simulates  the  task,  omits  part  of  the  training,  postpones 
training,  or  calls  someone  to  repair  the  equipment,  rate  ”3.” 


94.  Is  there  anything  unusual  about  the  LESSON  materials,  or  do  any  incidents 

occur  during  training  (that  are  not  covered  in  other  TEE  questions)  that  would 
interfere  with  learning? 

Describe  any  critical  incidents  on  your  worksheet. 

RATING:  If  anything  occurs  which  would  adversely  affect  student  learning, 

which  is  not  covered  in  other  TEE  questions,  rate  3. 
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1.  Is  the  TEST  FORMAT  appropriate 
for  the  OBJECTIVE? 

2.  Are  there  test  items  for  the  TLO 
or  all  of  its  critical  part$/LOs? 
(See  job  aid  for  critical  parts.) 


3.  Is  there  a  test  item  for  each 
critical  part  of  each  LO? 

(See  job  aid  for  critical  parts.) 


4.  Does  the  TASK  LEVEL  of  the 
test  item  match  the  TASK  LEVEL 
of  its  OBJECTIVE? 

3.  Does  the  content  of  the  test  item 
match  the  content  of  its 
OBJECTIVE? 

6.  Do  the  CONDITIONS  of  the  test 
item  match  the  CONDITIONS 

of  its  OBJECTIVE? 

7.  Do  the  STANDARDS  of  the  test 
item  match  the  STANDARDS  of 
its  OBJECTIVE? 

8.  For  true-false,  multiple  choice, 
and  matching  items  is  only  one 
answer  correct? 

9.  For  short  answer,  fill-in,  listing, 
and  performance  items  are  all 
acceptable  answers  in  the  answer 
key? 

10.  Does  the  test  item  provide 
opportunities  for  COMMON 
ERRORS  to  be  made? 

11.  Is  the  language  of  the  test  item 
easy  for  students  to  understand? 


12.  Is  the  test  item  different  from 
previous  PRACTICE  and 
EXAMPLES?  (USE-CONCEPT, 
USE-RULE,  or 
USE-PRINCIPLE  cnly) 


(See  table  in  job  aid.) 


1  =  Test  items  for  TLO  or  all 

parts/LOs 

2  s  No  test  items  for  TLO  and 

for  some  parts/LOs 

3  =  No  test  items  for  TLO  and 

for  most  parts/LOs 

1  =  Items  for  all  parts 

2  =  Items  for  many,  but  not  all  parts 

3  =  Items  for  only  a  few  parts  or 

for  no  parts 

(See  table  in  job  aid.) 


1  =  Same 

2  s  Slightly  different 

3  =  Very  different 

1  =  Exact  match 

2  =  Minor  mismatch 

3  =  Severe  mismatch 

1  a  Exact  match 

2  a  Minor  mismatch 

3  a  Severe  mismatch 

1  a  Only  one  answer  is  correct 
3  a  More  than  one  answer  can  be 
correct 

1  a  All  correct  answers  are  in 
answer  key 

3  a  Some  correct  answers  are  not 
in  answer  key 

I  =  Yes 
3  =  No 


1  a  Easy 

2  a  Somewhat  difficult 

3  a  Very  difficult 

1  a  Different 

2  =  Presented  before,  USE-UNAIDED 

3  a  Presented  before,  USE-AIDED 


13.  Is  the  answer  to  the  test  item 
given  away  by  other  item(s)? 


14*  Is  the  answer  to  the  test  item 
dependent  on  answering  previous 
item(s)  correctly? 


15.  Are  sketches  and  diagrams  used 
in  the  test  item  easy  to 
understand? 

16.  Is  the  test  item  tricky  or 
misleading? 


17.  Is  the  test  item  well  constructed? 
(See  job  aid  for  criteria  list  for 
the  test  format  used.) 

18.  When  performance  steps  are 
scored,  does  the  instructor  use 
a  checklist? 


19.  Is  each  correct  answer  position 
used  about  the  same  number  of 
times?  (true-false,  multiple 
choice,  or  matching  items  only) 

20.  Are  specific  patterns  of  correct 
answer  positions  repeated  across 
test  items  or  are  single  positions 
repeated  In  blocks?  (true-false, 
multiple  choice,  or  matching 
items  only) 

21.  Are  test  administration 
directions  complete? 


22.  Do  instructors  follow  the 
directions  when  administering 
the  test? 


23.  Arc  adequate  test  instructions 
provided  to  the  student? 


1  =  Answer  not  given  away 

2  =  Other  items  give  clues 

3  =  Answer  can  be  found  in  other 

item(s) 

1  =  Answer  not  dependent  on 
other  items 

3  =  Previous  items  must  be  correctly 
answered 

1  =  Easy  to  understand 

2  =  Somewhat  confusing 

3  a  Very  confusing 

1  =  Not  misleading 

2  =  Somewhat  misleading 

3  =  Very  misleading 

1  a  Meets  all  criteria 

2  a  Deficient  on  noncritical  criteria 

3  a  Deficient  on  critical  criteria 

1  a  Fills  in  completely 

2  a  Uses  as  a  reference  or  fills 

in  partially 

3  a  Does  not  use 

1  =  Yes 
3  a  No 


1  a  No  patterns  easily  seen 
3  a  Patterns  can  be  seen 


1  a  Directions  are  complete 

2  a  Directions  provided,  but 

incomplete  or  unclear 

3  a  Directions  are  not  provided 

1  a  Yes 

2  a  Sonr  variations  from  directions 

3  a  Sign,  cant  variations  from 

directions 

1  a  Yes 

2  a  Instructions  provided,  but 

unclear 

3  a  No  instructions  provided 
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24.  Does  the  FINAL  TEST  integrate 
tasks  as  they  are  integrated  in 
the  "real  world?”  . 

1  =  Yes 

2  s  Partially 

3  s  No,  tasks  are  tested  separately 

T 

25.  Are  tasks  and  task  steps  tested 
in  the  same  sequence  as  they  are 
performed  i:i  the  ’’real  world?" 

1  a  Yes 

2  s  Slightly  out  of  sequence 

3  a  Very  different  sequence 

26.  In  the  test  free  of  external  cues 
or  help? 

1  a  Yes 

2  s  Hints  given 

3  a  Answers  are  given  away  , 

27.  Are  motivational  techniques 
employed? 

1  a  Yes  , 

3  =  No 

28.  Is  the  trainee  attitude  positive? 

1  s  Positive 

2  =  Indifferent 

3  =  Hostile  or  frustrated 

29.  Are  course  ENTRY  SKILLS 
reviewed? 

1  s  Review  with  practice 

2  a  Review  with  no  practice 

3  =  No  review 

30.  Is  mastery  of  prerequisite  skills 
verified  prior  to  new  instructions’ 

1  =  Yes 

3  =  No 

31.  Are  OBJECTIVES  presented  to 
the  student? 

1  =  Yes 

3  a  No 

32.  Are  the  basic  PRESENTATION 
COMPONENTS  present? 

(See  guidance  and  tables  in 
handbook.) 

33.  Are  STATEMENTS  complete? 

1  =  STATEMENT  complete 

2  a  Few  parts  missing 

3  =  Many  parts  missing 

34.  Are  STATEMENTS  for  CONCEPTS, 
PROCEDURES,  or  RULES 
adequate?  (See  job  aid  criteria.; 

1  r  Completely  adequate 

3  a  Some  or  all  features  omitted 

35.  Does  STATEMENT  HELP  provide 
sufficient  explanation? 

1  a  Help  provides  sufficient 

explanation 

2  r  Help  gives  insufficient 

explanation 

3  a  Help  is  confusing 

36.  Does  training  include  instruction 
on  the  use  of  required  job 
performance  aids? 

1  a  Yes 

3  a  No 

37.  Are  EXAMPLES  and 

NON-EXAMPLES  adequate? 

I  a  Yes 

3  a  No 

B-4 

1 

_ 

38.  Is  EXAMPLE  HELP  adequate? 


39.  Are  EXAMPLES  sequences  from 
easy  to  hard?  (CONCEPTS  only) 

40.  Are  there  enough  EXAMPLES? 

(See  job  aid  for  criteria.) 

41.  Are  NON-EXAMPLES  included? 
(CONCEPTS  only) 

42.  Do  DEMONSTRATIONS  show 
how  to  correct /avoid  common 
errors? 

43.  Are  steps  In  a  DEMONSTRATION 
the  appropriate  size?  (See  job 
aid.) 

44.  Are  tasks  and  task  steps 
DEMONSTRATED  in  the  same 
sequences  as  they  are  performed 
in  the  "real  world?” 

45.  Are  memory  aids  used? 

(PRACTICE  REMEMBERING 
only) 

46.  Does  each  PRACTICE 
REMEMBERING  Item  have  the  same 
content  and  format  as  the  test 
Item? 


1  a  Help  provides  sufficient 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  a  Yes 
3  =  No 

1  =  Yes 
3  =  No 

1  a  Yes 
3  =  No 

1  =  Yes 
3  =  No 


1  =  Yes 

2  =  Step  size  is  too  small 

3  s  Step  size  is  too  large 

1  =  Yes 

2  *  Slightly  out  of  sequence 

3  a  Very  different  sequence 


1  a  Used 
3  a  Not  Used 


1  a  Same 

2  a  Same  content,  different 

format 

3  a  Different  content 


47.  Are  PRACTICE  USING  items  l  =  Yes 

sequenced  from  easy  to  hard?  3  a  No 

48.  Do  PRACTICE  USING  items  1  a  Yes 

provide  opportunities  for  3  «  No 


COMMON  ERRORS  to  be  made? 


49.  Are  PRACTICE  items  different  1  a  Different 

from  EXAMPLES?  (USE-CONCEPT,  3  a  Presented  before 

USE-RULE,  or  USE-PRINCIPLE 

only) 

50.  Does  PRACTICE  USING  integrate  1  a  Yes 

tasks  as  they  are  integrated  in  the  2  a  Partially 

"real  world?”  3  a  No,  tasks  are  practiced 

separately 


51.  Are  job  performance  aids 
(3PA$)  usable?  (See  criteria 
in  job  aid.) 

52.  Do  all  students  use  the  job 
performance  aid  (JPA)? 


53.  Does  the  TASK  LEVEL  of  the 
PRACTICE  item  match  that  of  the 
test  item(s)? 

54.  Does  the  CONTENT  TYPE  of 

the  PRACTICE  item  match  that  of 
the  test  item(s)? 

55.  Does  the  FORMAT  of  the 
PRACTICE  item  match  that  of 
the  test  item(s)? 

56.  Do  the  CONDITIONS  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

57.  Do  the  STANDARD  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

58.  Is  FINAL  PRACTICE  free  of 
external  cues  or  help? 


59.  Are  there  PRACTICE  items  for 
each  TLO  or  all  of  its  critical 
parts/LOs? 


60.  Is  there  a  PRACTICE  items  for 
each  critical  part  of  each  LO? 
(See  job  aid  for  critical  parts.) 


61.  Do  ail  students  PRACTICE? 


1  s  Easy  to  use 

2  =  Hard  to  use 

3  =  Unusable 

1  =  Yes 

2  =  Up  to  20%  do  not  use  JPA 

3  =  More  than  20%  do  not  use 

JPA 

1  =  Yes  * 

(2  and  3  —  See  table  in  job  aid.) 


1  -  Yes 
3  =  No 


1  =  Yes 

(2  and  3  —  See  table  in  job  aid.) 


1  =  Yes 

2  =  Slightly  different 

3  =  Very  different 

1  a  Yes 

2  a  Slightly  different 

3  =  Very  different 

1  a  Yes 

2  a  Hints  given 

3  a  Answers  are  given  away 

1  =  PRACTICE  items  for  the 

TLO  or  all  parts/LOs 

2  a  No  PRACTICE  items  for  the 

TLO  and  for  some  parts/LOs 

3  =  No  PRACTICE  items  for  the 

TLO  and  for  most  parts/LOs 

1  *  PRACTICE  for  all  parts 

2  a  PRACTICE  for  many,  but  not 

all  parts 

3  a  PRACTICE  for  only  a  few 

parts  or  for  no  parts 

1  a  Yes 

2  a  Up  to  20%  of  students  do  not 

PRACTICE 

3  a  More  than  20%  of  the  students 

do  not  PRACTICE 


62.  Do  ail  students  meet  the 

required  STANDARDS  in  FINAL 
PRACTICE? 


63.  Is  FEEDBACK  provided  for 
PRACTICE? 


64.  Is  FEEDBACK  HELP  adequate? 


65.  Is  TEAM  PRACTICE  provided? 


66.  Are  TEAM  PRACTICE 
CONDITIONS  the  same  as  (or  as 
close  as  possible  to)  those  of 
the  real  task? 

67,  Is  TEAM  PRACTICE  FEEDBACK 
provided? 


6S.  Is  FEEDBACK  HELP  for  TEAM 
PRACTICE  adequate? 


69.  Are  all  PRESENTATION 
COMPONENTS  separated  and 
identified? 

70.  Is  the  technical  quality  of  written 
or  spoken  material  adequate? 

(See  job  aid  for  criteria.  Make 
notes  on  specific  problems.) 

71.  Is  the  wording  of  written  or 
spoken  material  easy  for  the 
students  to  understand? 


72.  Is  the  instructor's  presentation 
or  the  narration  easy  to  listen  to? 


1  =  Yes 

2  s  Up  to  20%  of  students  do  not 

3  =  More  than  20%  of  students 

do  not 

1  =  FEEDBACK  HELP  is  given 

2  =  Correct  answer  only  l*  given 

3  =  No  feedback  is  given 

1  =  Help  gives  enough 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  a  Yes 
3  a  No 

1  a  Yes 

2  a  Slightly  different 

3  =  Very  different 


1  a  FEEDBACK  HELP  is  given 

2  a  Success/Failure  feedback 

only  is  given 

3  a  No  feedback  is  given 

1  a  Help  gives  enough  explanation 

2  a  Help  gives  insufficient 

explanation 

3  a  Help  is  confusing 

1  =  Yes 

2  a  Some  are  not 

3  =  Most  or  all  are  not 

1  a  Most  criteria  met 

2  a  Several  criteria  not  met 

3  a  Few  criteria  met 


1  a  Yes,  few  hard  words  and 

long  sentences 

2  a  Some  hard  words  and  long 

sentences 

3  a  Many  hard  words  and  long 

sentences 

1  a  Yes 

2  a  Dull  and  monotonous 

3  a  Hard  to  listen  to 


73.  Is  the  instructor’s  presentation 
or  the  narration  supported  by 
visuals? 

74.  Are  visuals  easily  understood? 


75.  Are  the  OBJECTIVES  (TLOs  and 
LOs)  within  each  LESSON 
sequenced  properly? 

(Prerequisites  taught  first.) 

76.  Are  the  LESSONS  sequenced 
properly  within  the  course? 

77.  Are  the  media  appropriate  for  the 
objectives?  (See  table  in  job  aid.) 


78.  Can  the  media  used  provide  ail 
necesary  stimuli? 

79.  Are  the  course  administration 
directions  complete? 


80.  Do  course  administration 
directions  make  realistic  demands 
of  students  and  instructors? 

81.  Is  the  instructor/trainee  ratio 
such  that  ail  students  can  see, 
hear,  and  receive  feedback? 


82.  Does  the  Instructor  follow  the 
methods  in  the  Instructor  Guide? 


83.  Docs  the  instructor  teach  all  of 
the  content  in  the  LESSON 
materials? 


84.  Did  the  instructor  limit  his 
teaching  to  the  content  in  the 
LESSON  materials? 


1  a  Completely 

2  =  Partially 

3  =  Not  at  all 

1  =  Yes 

2  =  Understandable  with  effort 

3  =  Very  hard  to  understand 

1  a  Yes 
3  =  No 


i  =  Yes 
3  a  No 

1  *  Yes 

3  s  No  (note  key  words, 
underlined  in  table,  on 
worksheet.) 

1  r  Yes 
3  a  No 

1  a  Yes 

2  a  Partially  incomplete 

3  a  Incomplete  or  non-existent 

1  a  All  demands  are  realistic 
3  a  Some  demands  are  unrealistic 
(Note  what  they  are.) 

1  a  Yes 

2  a  A  few  students  cannot  see, 

hear,  and  receive  feedback 

3  a  Many  students  cannot  see, 

hear,  and  receive  feedback 

1  a  Yes 

2  a  Follows  to  some  extent 

3  a  Follows  very  little  or 

not  at  all 

1  a  Yes 

2  a  Much  of  the  content 

3  a  Very  little  of  the  content 

(If  2  or  3,  note  what  was 
left  out.) 

1  a  Yes 

3  a  No  (Please  note  what  other 
things  he  taught.) 
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85.  Is  there  enough  space  for  ail 
of  the  trainees? 


86.  Is  instruction  free  of 
distractions? 


87.  Is  the  lighting  appropriate  for 
the  training  situation? 


88.  Is  the  temperature  appropriate 
for  the  training  situation? 


89.  Is  the  instructor’s  attitude 
positive? 

90.  Are  frequent  breaks  provided? 
(5-10  minute  breaks  every  hour) 


91.  Is  the  speed  of  presentation 
appropriate? 


92.  Was  the  allotted  training  time 
too  long  or  too  short? 


93.  Does  tlie  training  device/ 
equipment  used  in  training 
function  properly? 


1  &  Yes 

2  =  A  little  crowded 

3  =  Very  cramped  or  some 

students  can’t  fit  in  the 
space  at  all 

1  =  Yes 

2  s  Distractions  are  annoying 

3  =  Distractions  seriously 

interfere  with  the 
instruction 

1  «  Yes 

2  =  Students  have  trouble 

reading  or  seeing  displays 
&  equipment 

3  s  Students  cannot  read  or  see 

displays  &  equipment 

1  *  Yes 

2  a  Temperature  makes  students 

uncomfortable 

3  =  Temperature  seriously 

interferes  with  learning 

1  a  Yes 
3  a  No 

1  a  Yes 

2  a  Breaks  too  short  or 

infrequent 

3  a  Breaks  not  provided 

1  a  Yes 

2  a  Too  slow 

3  a  Too  fast 

1  a  Appropriate  length 

2  a  Too  long 

3  a  Too  short 

1  a  yes 

2  a  Minor  malfunctions,  little 

change  from  intended  task 
performance 

3  a  Major  malfunctions,  substantia! 

change  from  intended  task 
performance 


9$.  Is  there  anything  else  unusual  about  the  LESSON  materials,  or  do  any  other 
critical  incidents  occur  during  training  that  would  Interfere  with  learning? 
(Describe  each  one  below.  Rating  «  3) 
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Appendix  C 

JOB  PERFORMANCE  AID  FOR  USE  IN  PRODUCT  EVALUATION 

TEST  SECTION 

NOTE:  Before  proceeding  you  should  have  classified  each  test  item  and  its 

matching  OBJECTIVE  by  TASK  LEVEL  and  CONTENT  TYPE. 

On  the  following  questions,  rate  each  OBJECTIVE  before  going  on  to  the  next  objective: 

1.  Is  the  TEST  FORMAT  appropriate  for  the  OBJECTIVE? 

Appropriate  formats  are  shown  in  the  table  below: 

RATING:  (by  objective— TLO  and  LOs): 


If  Task  Level  Is: 

And  Content  Type  Is: 

And  Test  Format  Is: 

Then  Rate 

REMEMBER 

FACT  for  RECALL, 
CONCEPT,  PROCEDURE, 
RULE,  or  PRINCIPLE 

short  answer, 
fill-in  or 
listing 

1 

performance  (for  a 
later  USE-UNAIDED 
objective) 

2 

performance  (for  a 
later  USE-AIDED 
objective) 

3 

matching, 
true-false,  or 
multiple  choice 

2 

FACTS  FOR 

RECOGNITION 

matching, 
true-false,  or 
multiple  choice 

1 

performance 

3 

short  answer, 
fill-in,  or 
listing 

2 

USE-UNAIDED 

or 

USE-AIDED 

CONCEPT 

performance, 
matching, 
true-false, 
multiple  choice, 
short  answer,  or 
fill-in 

i 

listing 

2 

PRINCIPLE,  RULE, 
or  PROCEDURE 

performance, 
true-false 
multiple  choice, 
short  answer,  or 
fill-in 

1 

listing  or 
i  matching 

2 
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Are  there  test  items  for  the  TLO  or 
all  of  its  critical  parts/LOs? 


1 


=  Test  items  for  TLO  or  all 
parts/LOs 

2  =  No  test  items  for  TLO 
and  for  some  parts/LOs 

3  =  No  test  items  for  TLO 
and  for  most  parts/LOs 

NOTE:  As  defined  in  Section  4  of  the  Guidelines  for  Evaluating 

Objectives  in  Task  A4,  critical  parts/LOs  have  the  following 

characteristics: 

•  They  are  some  of  the  most  measurable  and  observable 
elements  of  the  task. 

•  They  have  serious  consequences  of  inadequate  performance 
in  terms  of  their  impact  on  task  performance. 

•  They  are  common  sources  of  failure. 

•  They  are  actions  which  the  soldiers  of  the  target 
population  do  not  already  know  how  to  do  and  must 
therefore  be  taught. 

NOTE:  For  entry  tests,  treat  the  top  level  of  entry  skills  as  TLOs, 


Is  there  a  test  item  for  each  critical  1  =  Items  for  all  parts 

part  of  each  LO?  2  =  Items  for  many,  but  not 

all  parts 

3  =  Items  for  only  a  few  parts 
or  for  no  parts 

The  critical  parts  of  LOs  are  defined  below. 

For  the  REMEMBER  TASK  LEVEL  each  piece  of  information  is  critical.  Each 
of  the  following  should  be  tested: 

•  Facts 

•  Concept  characteristics 

•  Procedural  steps 

•  Steps  and  conditions  for  branching  in  a  rule 

•  Guidelines  in  a  princple 

For  the  USE-AIDED  and  USE-UNAIDED  TASK  LEVELS,  the  parts  of  the 
OBJECTIVE’S  ACTION  are  critical: 

•  For  CONCEPTS,  are  all  CRITICAL  CHARCTERISTICS  tested^ 

Can  you  be  sure  the  student  knows  each  one  and  can  you 
identify  which  ones  he  does  not  know? 

•  For  PROCEDURES,  is  each  step  performed? 

•  For  RULES  and  PRINCIPLES,  is  each  step  performed?  Are  there 
items  for  all  possible  branches  or  decisions,  or  at  least  those 
that  would  commonly  occur  on  the  job? 


On  the  following  questions,  rate  each  test  item  before  going  to  the  next  item: 

4.  Does  the  TASK  LEVEL  of  the  test  item  match  the  TASK  LEVEL  of  its 
OBJECTIVE? 

Compare  the  ACTIONS  of  the  test  item  and  the  OBJECTIVE.  Record  the 
corresponding  rating  indicated  in  the  following  table. 

RATING  (by  item): 


Test  Item  TASK  LEVEL 


REMEMBER 

USE-UNAIDED 

USE-AIDED 

REMEMBER 

1 

2 

3 

OBJECTIVE 

TASK 

LEVEL 

USE-UNAIDED 

2 

1 

3 

USE-AIDED 

3 

2 

1 

5. 

Does  the  content  of  the  test  item 

1 

s 

Same 

match  the  content  of  its 

2 

= 

Slightly  different 

OBJECTIVE? 

3 

= 

Very  different 

6. 

Do  the  CONDITIONS  of  the  test 

1 

- 

Exact  match 

item  match  the  CONDITIONS 

2 

s 

Minor  mismatch 

of  its  OBJECTIVE? 

3 

55 

Severe  mismatch 

7. 

Do  the  STANDARDS  of  the  test 

1 

— 

Exact  match 

item  match  the  STANDARDS  of 

2 

Minor  mismatch 

its  OBJECTIVE9 

3 

= 

Severe  mismatch 

S. 

For  true-false,  multiple  choice, 

1 

= 

Only  one  answer  is 

and  matching  items  is  only  one 

correct. 

answer  correct? 

3 

— 

More  than  one  answer  can 
be  correct. 

9. 

For  short  answer,  fill-in,  listing, 

1 

- 

All  correct  answers  are  in 

and  performance  items  are  all 

answer  key. 

acceptable  answers  in  the  answer 

3 

= 

Some  correct  answers  are 

key? 

not  in  answer  key. 

10. 

Does  the  test  item  provided 

1 

- 

Yes 

opportunities  for  COMMON 

ERRORS  to  be  made? 

3 

— 

No 

11. 

Is  the  language  of  the  test  item 

1 

- 

Easy 

easy  for  students  to  understand? 

2 

= 

Somewhat  difficult 

3 

- 

Very  difficult 

12. 

Is  the  test  item  different  from 

1 

- 

Different 

previous  PRACTICE  and 

2 

r: 

Presented  before, 

EXAMPLES?  (USE-CONCEPT, 

USE-UNAIDED 
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USE-RULE,  or  USE-PRINCIPLE 

3 

= 

Presented  before, 

only) 

USE-AIDED 

13. 

Is  the  answer  to  the  test  item 

1 

5 

Answer  not  given  away. 

given  away  by  other  item(s)? 

2 

S 

Other  items  give  dues. 

3 

" 

Answer  can  be  found  in 
other  item(s). 

14. 

Is  the  answer  to  the  test  item 

1 

- 

Answer  not  dependent  on 

dependent  on  answering  previous 

other  items. 

item(s)  correctly? 

3 

— 

Previous  items  must  be 
correctly  answered. 

15. 

Are  sketches  and  diagrams  used 

1 

s 

Easy  to  understand 

in  the  test  item  easy  to 

2 

= 

Somewhat  confusing 

understand? 

3 

= 

Very  confusing 

16. 

Is  the  test  item  tricky  or 

1 

s 

Not  misleading 

misleading? 

2 

= 

Somewhat  misleading 

3 

= 

Very  misleading 

17. 

Is  the  test  item  well-constructed? 

1 

- 

Meets  all  criteria 

2 

= 

Deficient  on  non-critical 
criteria 

3 

s 

Deficient  on  critical 
criteria 

Select  the  appropriate  set  of  criteria  below  depending  upon  the  test  item 
format  being  used.  On  Worksheet  B1  write  the  name  of  the  test  format 
below  the  TEE  question  number.  Record  it  again  if  the  format  changes. 
Rate  the  item  as  indicated  below,  noting  the  letter(s)  corresponding  to  the 
identified  deficiency(ies)  in  the  "Description”  column.  Note  that  the  critical 
criteria  are  marked  with  asterisks  In  the  supplementary  guidance  to  this 
question. 


CRITERIA  FOR  WELL  CONSTRUCTED  TEST  ITEMS 


o  True-False 


*a. 

The  item 

false* 

*b. 

Negative 

c. 

The  item 

*d. 

The  item 

the  basis 

should  include  only  one  statement  (idea)  to  be  judged  true  or 

statements  and  double  negatives  should  not  be  used, 
should  be  short. 

should  not  be  so  obvious  that  it  can  be  answered  correctly  on 
of  common  sense  alone. 


*e.  Absolutes,  like  "none, 11  "never,"  "all,"  and  "every"  should  not  be  u;;ed. 

f.  The  Item  should  not  contain  the  words  "some,"  "any,"  or  "generally." 

*g.  The  item  should  deal  with  things  that  are  clearly  either  true  or  false. 


•  Multiple  Choice 

a.  All  alternatives  should  have  the  same  grammatical  structure. 

b.  The  item  stem  should  be  worded  positively,  that  is,  v/ords  like  "no"  and 
"not"  should  not  be  used. 

c.  Repetitive  phrases  should  be  placed  in  the  stem,  rather  than  in  the 
alternatives. 

d.  Numerical  choices  should  be  listed  in  order  of  magnitude. 

e.  The  item  should  not  require  a  great  deal  of  reading. 

*f.  Wording  of  the  stem  should  be  clear  and  umambiguous,  so  that  only  the 

one  correct  answer  is  possible. 

*g.  The  article  "a"  or  "an"  should  not  be  at  the  end  of  the  stem. 

*h.  All  alternatives  should  be  plausible  to  someone  who  doesn’t  know  the 

answer. 

1.  Alternatives  like  "A  and  B  only"  or  "All  of  the  above"  should  not  be 
used. 

*j.  Alternatives  should  be  approximately  the  same  length. 

*k.  Extreme  adjectives,  such  as  "complete,"  "total,"  and  "absolute"  should 
be  avoided. 


•  Matching 


a.  Directions  should  include  statements  of: 

•  the  contents  of  each  column 

•  the  basis  for  matching 

•  how  often  choices  may  be  used 

•  how  many  answers  are  possible  for  "givens" 

b.  All  choices  should  have  the  same  grammatical  structure. 

c.  Numerical  choice  should  be  in  order  of  magnitude. 

d.  The  item  should  not  .require  a  great  deal  of  reading. 
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e.  Choices  should  be  arranged  in  some  sensible  order,  for  example, 
alphabetically  or  logically. 

*f.  Unless  choices  can  be  used  more  than  once,  extra  entries  should  be 
included  in  the  choice  column. 

*g.  None  of  the  entries  in  either  column  should  appear  obviously  different 
from  each  other.  One  title  should  be  appropriate  for  all  entries  in  a 
column. 

*h.  Each  alternative  solution  should  be  a  plausible  answer  for  all  or  most 
of  the  problems. 

•  Fill-in 

a.  The  blank  should  be  at  or  near  the  end  of  the  sentence. 

*b.  One  and  only  one  word  or  phrase  should  correctly  complete  the  item, 

c.  Multiple  blanks  should  be  avoided. 

*d.  Blanks  should  require  key  words. 

*c.  Answers  should  be  specific  rather  than  general  in  nature. 

*f.  Items  should  measure  key  concepts,  not  nonessential  details. 

•  Short  answer 


a.  The  required  answer  should  be  short  (phrases  or  one  or  two  sentences). 

*b.  Directions  to  the  student  should  say  how  the  items  are  to  be  scored 

without  giving  clues  to  the  correct  answers. 

*c.  The  scoring  key  should  identify  allowable  synonyms  or  alternatives. 

*d.  The  question  should  be  complete  with  enough  information  for  the 

learner  to  generate  an  appropriate  answer. 

*e.  Credit  should  be  given  for  each  correct  part  of  an  answer. 

*f.  Questions  should  be  clear  so  that  all  learners  will  have  the  same 

Interpretation. 

*g.  A  model  answer  should  be  written  for  each  question  as  an  aid  to 
scoring,  or  a  checklist  of  attributes  of  a  good  answer  should  be 
provided  for  use  in  scoring. 


C-8 


h.  Items  should  be  assigned  points  according  to  their  vatue. 

i.  Point  values  should  be  made  known  before  the  test. 

j.  Items  should  be  scored  anonymously  (scorer  should  grade  students* 
papers  without  knowledge  of  their  identity). 

*k.  Content  should  be  scored,  not  grammar,  punctuation,  spelling,  and 
penmanship  (unless  specified  in  the  objective). 

1.  One  question  should  be  scored  across  all  students  before  preceeding  to 
the  next  one. 


Listing 

*a.  The  directions  should  specify  the  .number  of  things  to  be  listed,  (if 
appropriate  for  the  objective,  and  if  the  number  of  things  is  not  a 
clue.) 

*b.  The  directions  should  indicate  how  the  item  will  be  scored  without 
giving  clues  to  the  correct  answer. 

*c.  If  order  Is  important,  the  scoring  key  should  treat  sequence  separately. 

*d.  The  scoring  key  should  identify  allowable  synonyms  or  alternatives. 

*e.  Spelling  should  not  be  scored  unless  required  by  the  objective. 

f.  The  scoring  key  should  identify  relative  weights  of  different  items  on 
the  list,  if  appropriate. 


Performance 

NOTE:  A  performance  test  question  is  the  request  that  a  task  or  parts 

of  a  task  be  performed.  It  is  not  an  individual  point  on  a 
checklist. 

*a.  The  test  should  score  all  aspects  of  task  performance  and  qualites  of 
the  resulting  task  product  that  are  specified  in  the  objective  and  no 
others. 

*b.  The  diectlons  should  clearly  explain  what  the  student  is  to  do  and  how 
the  item  will  be  scored  (and/or  the  test  administrator  should  clearly 
communicate  that  to  the  student). 

Any  checklists  employed  should  follow  the  rules  in  c.  Sc  d.: 

c.  Each  step  should  be  self-explanatory  to  anyone  who  would  perform  the 
task. 


d.  Each  step  shot/  ’  hi  stated  as  a  single  action;  important  or  complex 
decisions  shoulc  e  presented  as  separate  steps. 

*e.  The  scoring  key  should  specify  all  criteria  the  performance  must  meet, 
such  as  completeness,  accuracy,  quality,  time  limit,  rate,  etc. 

f.  If  steps  in  the  performance  are  scored,  a  checklist  should  be  provided. 

*g.  Any  rating  scales  used  should  be  specific  enough  so  that  all  raters  will 

interpret  them  in  much  the  same  manner. 

*h.  All  raters  should  agree  on  judgments  of  rating  criteria.  For  example, 
all  raters  should  agree  that  given  steps  were  or  were  not  performed  or 
that  the  performance  was  fair,  good,  or  excellent. 

*i.  All  raters  should  agree  on  definitions  of  aspects  of  performance  and/or 
features  of  a  task  product. 


On  the  following  questions,  rate  each  test  as  a 

whole: 

19. 

Is  each  correct  answer  position  used 

I 

— 

Yes 

about  the  same  number  of  times? 

3 

- 

No 

(true-false,  multiple  choice,  or 
matching  items  only) 

20. 

Arc  specific  patterns  of  correct  answer 

1 

- 

No  patterns  easily  seen 

positions  repeated  across  test  items  or 
are  single  positions  repeated  In  blocks? 
(true-false,  multiple  choice,  or  matching 

3 

Patterns  can  be  seen 

items  only) 

21. 

Are  test  administration  directions 

1 

- 

Directions  are  complete 

complete? 

2 

S 

Directions  provided,  but 
Incomplete  or  unclear 

3 

— 

Directions  are  not 
provided 

23. 

Arc  adequate  test  instructions 

1 

- 

Yes 

provided  to  the  student? 

2 

s 

Instructions  provided,  but 

3 

s 

unclear 

No  instructions  provided 

24. 

Docs  the  FINAL  TEST  integrate 

l 

Yes 

tasks  as  they  are  integrated  in  the 

2 

s 

Partially 

"real  world?" 

3 

— 

No,  tasks  are  tested 
separately 

25. 

Are  tasks  and  task  steps  tested  in  the 

1 

- 

Yes 

same  sequence  as  they  arc  performed 

2 

s 

Slightly  out  of  sequence 

in  the  "real  world?" 

3 

— 

Very  different  sequence 
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PRESENTATION  SECTION 


NOTE:  Before  proceeding  you  should  have  classified  each  PRACTICE  item  and 

its  matching  OBJECTIVE  by  TASK  LEVEL  and  CONTENT  TYPE. 

On  the  following  questions  rate  each  PRESENTATION  COMPONENT: 


33.  Are  STATEMENTS  complete0 


1 

2 

3 


STATEMENTS  should  have  the  following  parts: 


STATEMENT  complete 
Few  parts  missing 
Many  parts  missing 


•  For  FACTS: 

•  For  CONCEPTS: 

•  For  PROCEDURES: 

•  For  RULES: 

•  For  PRINCIPLES: 


the  complete  fact  or  set  of  facts  should  be 
presented. 

all  critical  characteristics  and  how  they  are 
combined  should  be  given. 

all  steps  should  be  given  In  the  correct  order. 

all  steps  and  branching  decisions  should  be* 
given  in  the  correct  order. 

ail  potential  component  CONCEPTS, 
PROCEDURES,  RULES,  and/or  PRINCIPLES 
should  be  given  (unless  REMEMBERING  them 
can  be  considered  an  ENTRY  SKILL)  along 
with  the  PRINCIPLE  itself  (a  general  rule 
giving  typically  Incomplete  guidance  on  how 
to  construct  a  HIGHER  ORDER  RULE  for  the 
component  parts). 


34.  Arc  STATEMENTS  for  CONCEPTS, 
PROCEDURES,  or  RULES 
adequate? 


1  =  Completely  adequate 

3  a  Some  or  all  features 
omitted 


•  For  CONCEPTS:  a  decision  rule  or  search  strategy  for 

classifying  examples  and  non-examples  should 
be  given. 

•  For  PROCEDURES:  each  step  should  have  only  one  action. 


•  For  RULES: 


when  a  formula  is  used,  symbols  must  he 
defined. 


35.  Does  STATEMENT  HELP  provide 
sufficient  explanation? 


37.  Are  EXAMPLES  and 

NON-EXAMPLES  adequate0 
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1  s  Help  provides  suf  icient 

explanation 

2  s  Help  gives  insufficient 

explanation 

3  "  Help  Is  confusing 

1  3  Yes 

3  --  No 
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•  For  CONCEPTS: 


•  For  PROCEDURES, 
RULES,  or 
PRINCIPLES: 


EXAMPLES  should  show  all  CRITICAL 
CHARACTERISTICS;  for  NON-EXAMPLES  it 
should  be  clear  that  one  or  more  of  the 
CRITICAL  CHARACTERISTICS  is  missing. 


EXAMPLES  should  show  an  application  (use) 
of  the  PROCEDURE,  RULE,  or  PRINCIPLE. 


38.  Is  EXAMPLE  HELP  adequate? 


39.  Are  EXAMPLES  sequenced  from 
easy  to  hard?  (CONCEPTS  only) 


1  =  Help  provides  sufficient 

explanation 

2  =  Help  gives  insufficient 

explanation 

3  =  Help  is  confusing 

1  =  Yes 

2  =  No 


40.  Are  there  enough  EXAMPLES?  1  =  Yes 

3  =  No 


There  should  be  enough  EXAMPLES  to  cover  the  content  .rea  thoroughly. 

•  For  CONCEPTS:  At  leust  one  EXAMPLE  for  each  CRITICAL 

CHARACTERISTIC 


•  For  PROCEDURES 

or  RULES:  One  EXAMPLE  for  each  important  step 

EXAMPLES  for  all  possible  branches  or 
decisions 

•  For  PRINCIPLES:  At  least  one  EXAMPLE  for  each  major  type 

of  HIGHER  ORDER  RULE  that  should  be 
discovered 


RATING:  If  most  or  all  of  the  required  EXAMPLES  are  given,  rate  I. 

If  several  of  the  required  EXAMPLES  are  not  given,  rate  3. 


41. 

Are  NON-EXAMPLES  included? 

i 

£ 

Yes 

(CONCEPTS  only) 

3 

£ 

No 

42. 

Do  DEMONSTRATIONS  show 

1 

S 

Yes 

how  to  correct /avoid  common 

3 

S 

No 

errors? 

43. 

Are  -teps  In  a  DEMONSTRATION 

1 

- 

Yes 

the  appropriate  size9 

2 

s 

Step  size  is  too  small 

3 

5 

Step  size  Is  too  large 
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The  number  of  steps  or  concepts  that  are  demonstrated  to  trainees  at  one 
time  should  not  be  too  small  or  large. 


IF  TASK  INVOLVES: 

Fine  judgments 
Motor  Skills 
Difficult  concepts 
Unlearning  similar  concepts 

Straightforward  sequence 
Simple  concepts 


STEP-SIZE  SHOULD  BE: 

Relatively  SMALL 

Relatively  LARGE 


IF  TRAINEES  ARE: 

Very  new  to  the  task 
Low  academic  background 

Considerable  experience  in 
the  task 

High  academic  background 


STEP-SIZE  SHOULD  BE: 
Relatively  SMALL 

Relatively  LARGE 


45. 

Are  memory  aids  used? 

1 

9 

Used 

(PRACTICE  REMEMBERING 
only) 

3 

8 

Not  Used 

46. 

Does  each  PRACTICE 

I 

S 

Same 

REMEMBERING  item  have  the 
same  content  and  format  as  the 

2 

er 

Same  content,  different 
format 

test  item? 

3 

9 

Different  content 

47. 

Are  PRACTICE  USING  items 

1 

31 

Yes 

sequenced  from  easy  to  hard? 

3 

3 

No 

48. 

Do  PRACTICE  USING  items 

I 

s 

Yes 

provide  opportunities  for 

COMMON  ERRORS  to  be  made? 

3 

8 

No 

49. 

Arc  PRACTICE  items  different 

1 

3 

Different 

from  EXAMPLES?  (USE-CONCEPT, 
USE-RULE,  or  USE-PRINCIPLE 
only) 

3 

8 

Presented  before 

50. 

Dees  PRACTICE  USING  integrate 

I 

3 

Yes 

tasks  as  they  are  integrated  In  the 

2 

3 

Partially 

•’real  world?** 

3 

S 

No,  tasks  are  practiced 
separately 
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53.  Does  the  TASK  LEVEL  of  the 

PRACTICE  item  match  that  of  the 
test  item(s)? 

Compare  the  ACTIONS  of  the  PRACTICE  item  and  test  item(s)  that  go  with 
it. 

RATING: 


PRACTICE 

ITEM 

TASK 

LEVEL 


54.  Does  the  CONTENT  TYPE  of  1  =  Yes 

the  PRACTICE  item  match  that  of  3  =  No 

the  test  item(s)? 

55.  Does  the  FORMAT  of  the 
PRACTICE  item  match  that  of 
the  test  item(s)? 

RATING: 


Test  Item  TASK  LEVEL 


Test  Item  FORMAT 


Practice 

Item 

Format 
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56. 

Do  the  CONDITIONS  of  each 

1 

Yes 

FINAL  PRACTICE  item  match 

2 

= 

Slightly  different 

those  of  the  test  item(s)? 

3 

Very  different 

57. 

Do  the  STANDARDS  of  each 

1 

- 

Yes 

FINAL  PRACTICE  item  match 

2 

— 

Slightly  different 

those  of  the  test  item(s)? 

3 

- 

Very  different 

63. 

Is  FEEDBACK  provided  for 

1 

- 

FEEDBACK  HELP  is  given 

PRACTICE? 

2 

3 

Correct  answer  only  is 
given 

3 

3 

No  feedback  is  given 

64. 

Is  FEEDBACK  HELP  adequate*?  . 

1 

3 

Help  gives  enough 
explanation 

2 

= 

Help  gives  insufficient 
explanation 

3 

3 

Help  is  confusing 

65. 

Is  TEAM.  PRACTICE  provided? 

1 

- 

Yes 

3 

3 

No 

66. 

Are  TEAM  PRACTICE 

1 

_ 

Yes 

CONDITIONS  the  same  (or  as 

2 

3 

Slightly  different 

close  as  possible)  to  those  of 
the  real  task? 

3 

= 

Very  different 

67. 

Is  TEAM  PRACTICE  FEEDBACK 

1 

= 

FEEDBACK  HELP  is  given 

provided? 

2 

3 

Success/Failure  feedback 
only  is  given 

3 

3 

No  feedback  is  given 

68. 

Is  FEEDBACK  HELP  for  TEAM 

1 

- 

Help  gives  enough 

PRACTICE  adequate? 

2 

3 

explanation 

Help  gives  insufficient 
explanation 

3 

= 

Help  is  confusing 

On  the  following  questions  rate  each  Task/TLO  or  LO: 

32.  Are  the  basic  PRESENTATION 
COMPONENTS  present? 

RATING:  Rate  the  OBJECTIVE  in  accordance  with  the  tables  below.  The 

first  table  shows  the  required  components  for  each  task  level. 
The  second  table  gives  a  potential  rating. 


•  If  these  COMPONENTS  are  not  actually  present,  consider  them  missing 
unless: 

•  They  were  taught  within  the  past  few  days,  or 

•  They  were  taught  within  the  past  few  months  and  the  training 
materials  used  are  still  available  for  students  to  review,  or 

©  The  students  have  had  closely  related  MOS  experience. 


In  the  table  below,  select  the  row  within  the  appropriate  task  level  that 
matches  the  training  events  used  for  the  objective.  The  boxes  have  the 
following  meanings* 

—  Component  present 

Blank  Box  —  Component  missing 
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36.  Does  the  training  include  instruction  1  =  Yes 

on  the  use  of  required  job  performance  3  =  No 

aids? 

44.  Are  tasks  and  task  steps  1  =  Yes 

DEMONSTRATED  in  the  same  2  =  Slightly  out  of  sequence 

sequence  as  they  are  performed  in  3  =  Very  different  sequence 

the  "real  world?” 

51.  Are  JOB  PERFORMANCE  AIDS  1  =  Easy  to  use 

(JPAs)  usable?  2  =  Hard  to  use 

3  =  Unusable 

The  following  points  should  be  true  of  all  JPAs: 

•  Each  step  is  self-explanatory  to  anyone  who  would  perform  the  task. 

9  Each  step  picks  up  where  the  previous  step  left  off  (PROCEDURES, 
RULES,  and  PRINCIPLES  only). 

•  Illustrations  that  plainly  identify  components  to  be  maniuplated  or 
identified  are  included  where  applicable. 

59.  Are  there  PRACTICE  items  for  each  1  =  PRACTICE  items  for  the 

TLO  or  all  of  its  critical  parts/LOs?  TLO  or  all  parts/LO's 

2  =  No  PRACTICE  items  for 

the  TLO  and  for  some 
parts/LOs. 

3  =  No  PRACTICE  items  for 

the  TLO  and  for  most 
parts/LOs 

60.  Is  there  a  PRACTICE  item  for  each  1  =  PRACTICE  for  all  parts 

critical  part  of  each  LO?  2  s  PRACTICE  for  many,  but 

not  ail  parts 

3  *  PRACTICE  for  only  a  few 

parts  or  for  no  parts 

The  critical  parts  of  LOs  are  defined  beiow. 

For  the  REMEMBER  TASK  LEVEL  each  piece  of  information  is  critical.  Each 
of  the  following  should  be  tested: 

•  Facts 

•  Concept  characteristics 

a  Procedural  steps 

•  Steps  and  conditions  for  branching  in  a  rule 

•  Guidelines  in  a  principle 

For  the  USE-AIDED  and  USE-UNAIDED  TASK  LEVELS,  the  parts  of  the 
OBJECTIVES  ACTION  are  critical: 
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For  CONCEPTS,  are  all  CRITICAL  CHARACTERISTICS  tested?  Can 
you  be  sure  the  student  knows  each  one  and  can  you  identify  which 
ones  he  does  not  know? 

o  For  PROCEDURES,  is  each  of  the  steps  performed? 

o  For  RULES  and  PRINCIPLES,  is  each  of  the  steps  performed?  Are 

there  items  for  all  possible  branches  or  decisions,  or  at  least  those  that 
would  commonly  occur  on  the  job? 

77.  Are  the  media  appropriate  for  the  objectives? 

Compare  the  media  used  with  the  appropriate  media  in  the  table  below. 
Consider  groups  of  OBJECTIVES  that  fall  into  the  same  general  TASK  LEVEL 
and  CONTENT  LEVEL  as  defined  in  the  table. 

RATING:  If  the  media  are  appropriate,  rate  l. 

If  not,  rate  3  and  write  the  key  words  which  are  underlined  in 
f  the  right  hand  column  in  the  Description  column  of  your 

worksheet. 


If: 

j 

And  Task  Level/ 
(Content  Type) 

Is: 

Then 

Acceptable 

Media  Are:  a 

Problem  With 

Mismatch 

Consequences 
of  task  error 
are  serious,  b 

USE 

Large  Equipment 
Portable  Equipment 
Simulator 

(Required  for  final 
practice  in  addition 
to  other  media  below 
used  earlier  in 
training.) 

No  real  practice.  Other 
media  cannot  provide 
the  realism  needed  to 
PRACTICE  the  task 
adequately  under  these 
conditions 

Self-Instruction 
(the  course 
manager  does 
not  give 
FEEDBACK  c 
to  students) 

REMEMBER 

Audio 

Filmstrip 

Motion  Picture 

Printed  Text 

Slide/Tape 

Training  Aid 

TV  Cassette 

Computer 

Programmed  Text 
Interactive  TV 

Training  Device 
(supplemented  with 
printed  text) 

Portable  Equipment 
(supplemented  with 
printed  text) 

Incomplete  Information. 

Other  media  would  not 
typically  present  enough 
information  without  the 
aid  o i  an  instructor. 
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If: 

And  Task  Level/ 
(Content  Type) 

Is: 

Then 

Acceptable 
Media  Are:  a 

Problem  With 
Mismatch 

USE/MENTAL 
SKILLS  d 

Computer 

Programmd  Text 
Interactive  TV 
Training  Device 
Motion  Picture 
Slide/Tape 

TV  Cassette 

Filmstrip 

Audio 

No  FEEDBACK.  Other 
media  typically  cannot 
provide  corrective 
FEEDBACK  without 
an  instructor. 

USE/MOTOR 

SKILLS 

Training  Device 
(supplemented  with 
printed  text) 

Portable  Equipment 
(supplemented  with 
printed  text) 

No  hands-on  practice. 
Other  media  cannot 
provide  the  necessary 
motor  manipulations 

Instructor-led 

instruction 

REMEMBER 

Instructor  with  any 
other  media 

"I”ninr 

USE/MENTAL 
SKILLS  d 

Instructor  with  any 
other  media 

— 

USE/MOTOR 
SKILLS  e 

Instructor  with: 
Portable  Equipment, 
Large  Equipment, 
Training  Device, 
or  Simulator 

No  hands-on  practice. 
Other  media  cannot 
provide  the  necessary 
motor  manipulations 

a  Media  definitions  may  be  found  in  Appendix  E  at  the  end  of  these  guidelines. 

k  When  the  graduate  is  first  required  to  perform  the  task  on  the  job,  are  the 
consequences  of  error  serious? 

c  A  course  manager  would  do  no  more  than  grade  tests.  He  would  not  give 
FEEDBACK  on  PRACTICE.  The  media  must  do  that  in  this  sctt’ng. 

d  Mental  skills  arc  CONCEPTS,  PROCEDURES,  RULES,  or  PRINCIPLES  which  are 
performed  primarily  in  the  student's  mind.  A  skill  may  have  both  mental  and 
motor  skills  associated  with  it,  but  if  the  motor  skills  are  so  simple  that  anyone 
who  knew  the  mental  skills  could  perform  them  without  training,  the  overall  skill 
should  be  considered  a  mental  skill.  Sometimes  a  TLO  is  broken  down  into 
mental  and  motor  LOs. 

e  Motor  skills  are  PROCEDURES  which  require  the  smooth  timing  of  muscular 
movements.  All  motor  skills  have  a  mental  component,  but  with  motor  skills  the 
physical  activity  must  be  practiced  in  order  to  achieve  proficiency. 
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NOTE:  All  of  the  media  shown  for  each  situation  are  not  necessarily  the  best 

media  in  terms  of  costs  and  practicality;  however  they  can  provide  the 
necessary  learning  characteristics  for  those  conditions. 


78. 

Can  the  media  used  provide  all 

1 

= 

Yes 

necessary  stimuli? 

3 

= 

No 

On  the  following  questions  rate  each  LESSON: 

27. 

Are  motivational  techniques  employed? 

I 

= 

Yes  (Please  describe 
them.) 

3 

= 

No 

30. 

Is  mastery  of  prerequisite  skills 

1 

- 

Yes 

verified  prior  to  new  instruction? 

3 

r 

No 

31. 

Are  OBJECTIVES  presented  to  the 

1 

s 

Yes 

student? 

3 

s 

No 

69. 

Are  all  PRESENTATION 

1 

_ 

Yes 

COMPONENTS  separated  and 

2 

S 

Some  are  not 

identified? 

3 

= 

Most  or  all  are  not 

70. 

Is  the  technical  quality  of  written 

1 

Most  criteria  met 

or  spoken  material  adequate? 

2 

= 

Several  criteria  not  met 

3 

s 

Few  criteria  met 

Whether  text  appears  in  print,  in  some  other  visual  medium,  as  narration,  or 
as  spoken  in  a  lecture  or  demonstration,  it  should  meet  the  following 
standards: 

a.  Each  COMPONENT  should  be  directed  at  the  primary  audience. 

b.  The  instruction  should  be  performance-oriented  rather  than  topic 
oriented.  It  should  tell  "what  to  do”  or  "how  to  do  it,"  rather  than 
telling  about  a  task. 

c.  Main  points  should  stand  out.  They  should  not  get  lost  in  detail,  so 
that  secondary  points  appear  equal  to  main  points.  COMPONENTS 
shoulld  not  be  too  wordy,  either  because  there  is  too  much  detail  (more 
than  the  essentials  needed  to  perform  the  job),  or  because  material  is 
redundant  (with  information  being  needlessly  repeated). 

d.  Information  should  be  presented  in  an  orderly  manner  so  that  it  is  not 
confusing.  Check  for: 

•  Redundancy  between  paragraphs  or  ideas 

•  Missing  information 

•  Scattered  information— information  about  the  same  main  point  not 
in  one  place 
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•  Remote  references— references  to  text  or  illustrations  in  other 
places 

e.  Visual  aids  should  communicate  something.  They  must  function  in  one 
or  more  of  the  following  ways.* 

•  Substitute  for  text  or  narration 

•  Lighten  the  burden  of  detail  in  text  or  narration 

•  Reinforce  or  summarize  information  in  text  or  narration 

f.  Is  there  a  run-together  format?  A  solid  mass  of  print  should  be 
"chunked*  visually  so  that  separate  points  can  be  easiiy  identified. 
Items  in  A/V  presentations  or  lectures  should  also  be  distinguishable. 

g.  Perspective— In  visual  media  do  visuals  provide  the  viewer  with  an 
adequate  frame  of  reference  with  which  to  judge  size  and  distance? 


RATING:  Considering  the  criteria  listed  above,  if  most  or  all  of  the 

applicable  criteria  are  met,  rate  1. 

If  several  applicable  criteria  are  not  met,  rate  2. 


If  few  applicable  criteria  are  met,  rate  3. 

NOTE:  (1)  Be  sure  to  note  what  the  specific  problems  are  in  the 

Description  column  of  your  worksheet. 

(2)  You  should  make  this  rating  for  an  entire  LESSON  unless 
material  for  certain  OBJECTIVES  merits  different  ratings. 


71.  Is  the  wording  of  written  or  spoken 
material  easy  for  the  student  to 
understand? 


72.  Is  the  instructor's  presentation  or 
the  narration  easy  to  listen  to? 


1  =  Yes,  few  hard  words  and 

long  sentences 

2  =  Some  hard  words  and 

long  sentences 

3  9  Many  hard  words  and 

long  sentences 

1  b  Yes 

2  r  Dull  and  monotonous 

3  s  Hard  to  listen  to 


73.  Is  the  instructor's  presentation,  the 
narration  or  text  materials 
supported  by  visuals? 

74.  Are  visuals  easily  understood? 


75.  Are  the  OBJECTIVES  (TLOs  and 
LOs)  within  each  LESSON 
sequenced  properly?  (Prerequisites 
taught  first.) 


1  s  Completely 

2  =  Partially 

3  8  Not  at  all 

1  8  Yes 

2  a  Understandable  with  effort 

3  s  Hard  to  understand 

I  *  Yes 
3  =  No 
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91.  Is  the  speed  of  presentation  1  =  Yes 

appropriate?  2  =  Too  slow 

3  =  Too  fast 

92.  Was  the  allotted  training  time  too  1  =  Appropriate  length 

short  or  too  long?  2  =  Too  long 

3  =  Too  short 


On  the  following  questions  make  one  rating  for  the  course: 

29.  Are  course  ENTRY  SKILLS  reviewed?  1  =  Review  with  practice 

2  =  Review  with  no  practice 

3  s  No  review 

76.  Are  the  LESSONS  sequenced  properly 
within  the  course? 

LESSONS  that  teach  tasks  which  are  SUBORDINATE  SKILLS  to  tasks  taught 
in  other  LESSONS  should  be  taught  first. 

Look  at  each  LESSON  starting  at  the  beginning  of  the  course  and  ask  what 
the  students  must  already  know  in  order  to  learn  the  TASKS  in  the  LESSON 
(the  LESSON  prerequisites). 

RATING:  If  the  LESSON  prerequisites  were  taught  in  previous  LESSONS  or 

are  ENTRY  SKILLS,  rate  1. 

If  not,  rate  3  and  note  what  knowledge  or  skills  should  hav? 
been  taught  earlier  in  the  Description  column  of  your  worksheet. 

79.  Are  the  course  administration  1  =  Yes 

directions  complete?  2  =  Partially  incomplete 

3  =  Incomplete  or  non-existent 

SO.  Do  course  administration  directions  i  =  All  demands  are  realistic 

make  realistic  demands  of  students  3  =  Some  demands  are 

and  instructors?  unrealistic.  (Note  what 

they  are.) 

99.  Is  there  anything  else  unusual  about  the  LESSON  materials  that  would 
Interfere  with  learning?  (Describe  each  one:  Rating  *  3.) 
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Appendix  D 

JOB  PERFORMANCE  AID  FOR  USE  IN  PROCESS  EVALUATION 


Are  adequate  test  instructions  provided  to  the  student? 

Two  types  of  instructions  may  be  required: 

1.  Item  instruction  for  groups  of  test  items.  These  should  be  clear  and 
should  provide  enough  information  to  answer  the  items  correctly.  (Be 
sure  to  indicate  which  items  the  rating  applies  to  on  your  worksheet.) 

2.  General  instructions  shoutd  state: 

•  the  test's  purpose 

•  any  time  limits 

•  descriptions  of  test  conditions 

•  descriptions  of  test  standards 

•  how  to  respond  to  each  item 

•  general  regulations 

There  should  be  no  question  in  a  student's  mind  as  to  how  to  take  tlw 
test  or  how  to  behave  during  the  test. 

RATING:  If  yes9  l.e..»  adequate  test  instructions  to  the  student  are 

provided,  rate  1. 

If  instructions  are  provided  but  are  not  completely  clear,  rate  2. 
If  no  instructions  arc  provided,  rate  3. 


Are  job  performance  aids  (JPAs)  usable? 

The  following  points  should  be  true  of  all  JPAs: 

•  Each  step  is  self-explanatory  to  anyone  who  would  perform  the  task* 

•  Each  step  picks  up  where  the  previous  step  left  off. 

•  Illustrations  that  plainly  identify  components  to  be  manipulated  or 
identified  are  included  where  applicable. 

RATING:  If  the  job  aid  is  easy  to  use,  rate  1. 

If  the  job  aid  is  hard  to  use,  rate  2. 

If  the  Job  aid  is  unusable,  rate  3. 
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53.  Docs  the  TASK/LEVEL  of  the  PRACTICE  ITEM  match  that  of  the  test  item(s)? 
Compare  the  ACTIONS  of  the  PRACTICE  item  and  test  Item(s)  that  go  with  it. 
RATING: 


PRACTICE 

ITEM 

TASK 

LEVEL 


Test  Item  TASK  LEVEL 


REMEMBER 

USE -UNAIDED 

USE-AIDED 

REMEMBER 

1 

2 

2 

USE-UNAIDED 

2 

I 

2 

USE-AIDED 

3 

3 

I 

55.  Does  the  format  of  the  PRACTICE  item  match  that  of  the  test  item? 
RATING: 


Test  Item  FORMAT 


Practice 

Item 

Format 


True/ 

False 

Multiple 

Choice 

Matching 

Fill-In 

Short 

Answer 

Listing 

Performance 

True/ 

False 

1 

2 

2 

2 

2 

2 

2 

Multiple 

Choice 

2 

1 

2 

2 

2 

2 

3 

Matching 

2 

2 

1 

2 

2 

2 

3 

Fill-In 

2 

2 

2 

1 

2 

2 

2 

Short 

Answer 

2 

2 

2 

2 

1 

2 

2 

Listing 

2 

2 

2 

2 

2 

1 

2 

Perfor¬ 

mance 

2 

2 

2 

2 

2 

2 

| 

1 

70.  Is  the  technical  quality  of  written  or  spoken  material  adequate? 

Whether  text  appears  in  print,  in  some  other  visual  medium,  as  narration,  or 
as  spoken  in  a  lecture  or  demonstration,  it  shuild  meet  the  following 
standards: 

a.  Each  COMPONENT  should  be  directed  at  the  primary  audience. 
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b.  The  instruction  should  be  performance-oriented  rather  than  topic 
oriented.  It  should  tell  "what  to  do”  or  "how  to  do  it,"  rather  than 
telling  about  a  task. 

c.  Main  points  should  stand  out.  They  should  not  get  lost  in  detail,  so 
that  secondary  points  appear  equal  to  main  points.  COMPONENTS 
should  not  be  too  wordy,  either  because  there  is  too  much  detail  (more 
than  the  essentials  needed  to  perform  the  job),  or  because  material  is 
redundant  (with  information  being  needlessly  repeated). 

d.  Information  should  be  presented  in  an  orderly  manner  so  that  it  is  not 
confusing.  Check  for: 

•  Redundancy  between  paragraphs  or  ideas 

•  Missing  information 

a  Scattered  information— information  about  the  same  main  point  not 
in  one  place 

•  Remote  references— references  to  text  or  illustrations  in  other 
places 

e.  Visual  aids  should  communicate  something.  They  must  function  In  one 
or  more  of  the  following  ways? 

•  Substitute  for  text  or  narration 

•  Lighten  the  burden  of  detail  in  text  or  narration 

•  Reinforce  or  summarize  information  in  text  or  narration 

f.  Is  there  a  rurvtogether  format?  A  solid  mass  of  print  should  be 
"chunked"  visually  so  that  separate  points  can  be  easily  Identified. 

Items  in  A/V  presentations  or  lectures  should  also  be  distinguishable. 

g.  Perspective— In  visual  media  do  visuals  provide  the  viewer  with  an 
adequate  frame  of  reference  with  which  to  judge  size  and  distance? 

RATING:  Considering  the  criteria  listed  above,  if  most  or  all  of  the 

applicable  criteria  are  met,  rate  1. 

If  several  applicable  criteria  are  not  met,  rate  2. 

If  few  applicable  criteria  are  met,  rate  3. 

NOTE:  (l)  Be  sure  to  note  what  the  specific  problems  are  in  the 

Description  column  of  your  worksheet. 

(2)  You  should  make*  this  rating  for  an  entire  LESSON  unless 
material  for  certain  OBJECTIVES  merits  different  ratings. 
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Is  the  instructors  attitude  positive? 

Does  the  instructor  behave  in  a  manner  that  enhances  the  students* 
motivation  for  learning? 

•  Does  the  instructor  encourage  questions? 

o  Is  he  helpful? 

•  Respectful  of  the  trainees? 

•  Tolerant  of  failure? 

•  Available  for  assistance? 

RATING:  If  the  instructor’s  attitude  is  positive,  rate  J. 


If  not,  rate  3. 
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1.  Audio.  Equipment  which  delivers  a  pre-recorded  audio  message,  which  may  be 
amplified  for  groups.  The  device  involved  may  also  have  recording  capability. 
Audio  systems  include  audio  tape  systems,  either  cassette  or  reel-to-reel; 
stylus-  or  laser-read  disc  systems;  and  computer-generated  audio. 

2.  Chart.  A  flat  surface  for  presenting  displays  to  a  class,  containing  messages 
in  print,  handwriting,  drawings,  pictures,  graphs,  or  other  forms.  A 
blackboard  is  included  in  the  chart  category. 

3.  Computer.  The  computer  displays  printed  words,  symbols,  and  diagrams,  and 
may  accept  keyboard,  light  pen,  or  touch  inputs  as  student  responses. 

Suitably  programmed,  it  can  provide  instruction  adapted  to  the  individual 
students  state  of  knowledge,  display  corrective  feedback,  and  perform  other 
adaptive  and  interactive  operations.  New  varieties  of  instructional  use  are 
foreseen  as  auditory  inputs  and  outputs  become  available. 

4.  Filmstrip.  Projection  of  pictures  which  are  sequentially  printed  on  a  strip  of 
film.  Such  projection  may  or  may  not  be  synchronized  with  an  audio  message 
from  a  tape  player.  The  function  of  this  equipment  is  the  same  as  that  of 
slide  projection,  with  and  without  synchronized  audio.  It  is,  however,  a 
separate  medium,  because  it  requires  different  materials  and  equipment  and 
cannot  be  readily  re-sequenced  as  slides  can. 

5.  Instructor.  As  a  source  of  instruction,  the  most  versatile  medium  is  an 
instructor.  Four  important  functions  of  an  instructor  ares  (a)  delivery  of 
audio  messages,  (b)  directing  individual  student  attention  to  particular  features 
of  a  display;  and  (c)  human  modeling,  by  serving  as  a  model  for  choices  of 
personal  action;  and  (d)  providing  corrective  feedback  to  the  individual 
student.  Types  of  individuals  that  could  fall  into  this  category  would  be 
instructors,  who  can  perform  any  of  these  functions,  and  assistant  instructors, 
who  typically  perform  limited  functions,  such  as  providing  feedback. 

6.  Interactive  TV.  This  medium  combines  a  microcomputer  with  videodisc  or 
videotape  playback  equipment.  An  interactive  TV  program,  with  sound,  can 
become  a  highly  useful  "teaching  machine"  since  its  branching  capabilities  can 
provide  feedback  tailored  to  the  individual  student.  It  is  different  from  a 
computer  in  that  it  can  display  motion-visuals  and  realistic  still-visuals. 

7.  Lar^e  Equipment.  Operating  equipment  that  is  large  and  non-portable,  such 
as  a  helicopter,  truck,  or  artillery  piece. 

8.  Motion  Picture.  A  film  projection  system  displaying  moving  pictures  on  a 
screen,  often  accompanied  by  a  synchronized  audio  track.  Some  systems 
permit  large  screen  projection  to  large  groups.  Systems  in  this  medium 
include  8mm  projector  systems  (cither  regular  or  super  S),  as  well  as  16mm 
and  33mm  systems.  Included  in  this  category  are  both  sound  and  silent 
projection  systems  although  the  latter  is  infrequently  used. 
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9.  Overhead  Projection*  Projection  of  transparencies  (about  8  by  10  inches) 
displaying  print,  pictures,  or  diagrams  onto  a  nearby  screen.  Overhead 
projection  and  the  changing  of  transparencies  may  be  done  by  the  instructor 
at  the  front  of  a  classroom  or  by  using  rear-screen  projection.  This  category 
also  includes  the  opaque  projector. 

10.  Portable  Equipment.  A  unitary  device  (rifle,  protective  mask),  or  a 
component  of  a  real  system  (dial,  carburetor),  which  is  small  enough  to  be 
readily  lifted  and  transported.  Portable  equipment  is  often  the  medium  of 
choice  whenever  direct  practice  of  task  performance  is  possible  and  desirable. 
Such  equipment  may  sometimes  have  a  specially  built-in  mechanism  to  provide 
corrective  feedback  to  students,  and  thus  may  function  in  the  manner  of  a 
training  device. 

11.  Printed  Text.  Pages,  cards,  or  other  surfaces  containing  meaningful  materials 
in  printed  form,  for  use  by  individual  students.  It  is  always  possible  for 
printed  text  to  contain  printed  pictures  or  diagrams.  Microfiche,  microfilm, 
and  other  such  reduced  print  media  will  be  included  in  this  category,  even 
though  they  require  additional  equipment  for  their  use. 

12.  Programmed  Text.  This  medium  is  the  basic  form  of  programmed  instruction, 
using  printed  cards  or  pages,  usually  collected  into  a  booklet.  A  printed  text 
(see  item  11)  takes  on  a  programmed  form  when  it  requires  frequent 
responses  and  provides  feedback. 

13.  Simulator.  Typically,  equipment  used  for  training  or  proficiency  maintenance 
which  reproduces  many  of  the  operating  characteristics  of  the  real  equipment. 
An  example  is  an  aircraft  flight  simulator. 

14.  Slides.  A  medium  without  audio  which  projects  still  pictures  from  small 
slides  onto  a  variety  of  possible  screens,  ranging  from  inches  to  yards  across. 

13.  Slide/Tape.  Projection  of  still  pictures  from  small  slides,  accompanied  by 
synchronoizcd  audio-tape  messages.  This  category  may  include  a  combined 
slide  projector/audio  tape  system  with  automatic  cueing  or  separate  slide  and 
audio  tape  machines  with  synchronization  provided  through  direct  human 
assistance.  Slide/tape  systems  may  involve  more  than  one  projector. 

Although  the  Bessler  Cue-See  can  provide  motion  visuals,  it  is  included  in  the 
slide/tape  category  because  its  typical  use  is  closer  to  slide/tape  than  it  is  to 
motion  pictures. 

16.  Training  Aid.  A  surface  layout,  model,  or  mockup,  providing  a  display  of 
parts  and  processes  of  a  system  on  which  instruction  is  being  given.  The 
simplest  form  of  a  training  aid  is  a  chart.  However,  training  aids  may 
display  part  of  real  equipment  (rather  than  pictures),  and  even  provide 
dynamic  views  of  processes  (as  in  a  working  model  of  a  hydraulic  system). 

In  contrast  to  training  device  (see  item  17),  a  training  aid  makes  no  specific 
provision  for  corrective  feedback  to  the  student. 

17.  Training  Device.  Equipment  on  which  mental  skills  procedures  or  motor  skills 
can  be  practiced,  and  which  provide  corrective  feedback  to  the  student.  (A 
training  aid  does  not  automatically  provide  corrective  feedback.)  A  training 
device  has  its  own  operating  principles  u.td  does  not  attempt  to  reproduce 
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those  of  the  real  equipment,  as  does  a  simulator.  An  example  of  a  training 
device  is  a  Link  trainer, 

TV  Cassette.  A  device  which  makes  possible  the  display  of  a  pre-recorded 
program  (picture  and  sound)  on  a  TV  receiver.  The  device  may  also  have  a 
recording  capability.  Delivery  systems  covered  by  this  category  include  reel- 
to-reel  videotape  systems,  videocassette  systems,  and  videodisc  playback 
systems. 
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TRAINING  MATERIALS  FOR  USE  BY  THE  TEE  ANALYST  AND 
ASSOCIATE  ANALYST  ON  TASK  LEVEL  AND  CONTENT  TYPE 

SELF-INSTRUCTION  ON  TASK  LEVEL  AND  CONTENT  TYPE 


Study  the  material  on  the  next  eight  pages  and  read  the  directions  on  the  page 
following  them  titled  "Classification  Practice."  Then  do  the  practice  items  which 
follow  it.  Take  the  seif- test  in  the  next  section  and  compare  your  answers  to  the 
Feedback  after  that.  If  you  still  feel  uncertain  of  your  ability  to  classify  task  level 
or  content  type,  classify  the  objectives  in  the  remaining  section  and  compare  your 
answers  to  the  feedback  there. 


THE  CLASSIFICATION  SYSTEM: 

The  following  classification  system  is  used  in  all  TEE  procedures.  It  is 
applied  to  the  three  main  parts  of  instruction:  objectives,  tests,  and 
instructional  presentations. 

Each  objective,  test  item,  or  piece  of  presentation,  can  be  classified 
according  to: 

1.  What  the  student  must  do,  i.e.,  the  TASK  to  be  performed,  and 

2.  The  type  of  information  the  student  must  learn,  he.,  the 
instructional  CONTENT. 


In  the  TEE  guidelines,  these  two  classification  dimensions  have  been  combined 
to  form  the  TASK/CONTENT  MATRIX. 


THE  TASK  DIMENSION: 

There  are  two  main  TASKS  a  student  can  perform: 

1.  He  can  REMEMBER  information,  or 

2.  He  can  USE  the  information  to  do  something. 
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EXAMPLE: 


Here  are  two  test  items: 

1.  The  symbol  for  resistor  is _ . 

2.  Using  your  knowledge  of  electronic  theory,  what  would  happen  in 
the  circuit  shown  below  if  the  load  resistance  were  shorted? 

These  two  test  items  differ  with  respect  to  what  the  student  is  supposed  to 
do  (TASK).  In  number  1,  the  student  has  to  REMEMBER  something,  and  in 
number  2,  the  student  has  to  apply  or  USE  his  knowledge  in  a  new  situation. 


THE  CONTENT  DIMENSION: 

There  are  five  types  of  CONTENT: 


FACT 

CONCEPT 

PROCEDURE 

RULE 

PRINCIPLE 

REMEMBER 

USE 

Not 

Possible 

FACTS  are  simple  associations  between  names,  objects,  symbols,  locations, 
etc. 

CONCEPTS  are  categories  or  classifications  defined  by  certain  critical 
characteristics. 

PROCEDURES  consist  of  ordered  sequences  of  steps  or  operations  performed 
on  a  single  object  or  in  a  specific  situation. 

RULES  also  consist  of  ordered  sequences  of  operation,  but  can  be  performed 
on  a  variety  of  objects  or  in  a  variety  of  situations. 

PRINCIPLES  are  general  rules  which  are  used  to  select  or  create  other 
principles,  rules,  procedures,  or  concepts  and  apply  them  to  solve  problems, 
he.,  describe  why  or  how  things  happen  or  what  will  happen  in  a  given 
situation. 

EXAMPLES: 

The  following  examples  illustrate  the  five  content  areas  for  the  REMEMBER 
task  level: 


F-3 


REMEMBER  FACT 

1. 

The  symbol  for  resistor  is 

2. 

List  the  names  of  the  parts  in  the  wind- 

indicating  instrument. 

REMEMBER  CONCEPT 

1. 

List  the  critical  characteristics  of  a  jet 
pump. 

2. 

Define  the  various  kinds  of  clouds  (cumulus, 
stratus,  etc.). 

REMEMBER  PROCEDURE 

1. 

•  List  in  order  the  steps  for  cleaning  an  M-16 
rifle. 

2. 

Describe  the  procedure  for  preparing  and 
sending  a  radio  message. 

REMEMBER  RULE 

1. 

List  the  steps  involved  in  calculating 
magnetic  heading,  given  true  heading 
magnetic  variation,  and  compass  deviation. 

2. 

State  the  general  rule  for  solving  for  circuit 
current,  given  voltage  and  resistance. 

REMEMBER  PRINCIPLE 

1. 

State  the  principles  of  leadership. 

2. 

Recall  the  reasons  why  hydraulic  fluid 
contamination  must  be  avoided. 

NOTE:  FACTS  can  only  be  remembered,  but  lor  the  other  content  types,  the 

student  may  be  asked  to  USE  his  knowledge  to  classify,  perform,  solve 
or  predict. 

The  following  are  examples  of  the  USE  task  level  for  all  content  types 
except  FACTS: 


USE  CONCEPT 

1. 

Which  of  the  pumps  are  jet  pumps? 

2. 

Given  photographs  of  clouds,  sort  them 
according  to  type  (cumulus,  stratus,  etc.) 

USE  PROCEDURE 

1. 

Clean  an  M-16  rifle. 

2. 

Prepare  and  send  a  radio  message. 

USE  RULE 

1. 

Calculate  the  magnetic  heading  from  the  two 
heading  magnetic  variations,  and  compass 
deviation. 

2. 

Given  the  values  for  voltage  and  resistance, 
calculate  the  current  flow. 

USE  PRINCIPLE 

1. 

Describe  several  potential  courses  of  action  in 
a  given  management  situation  and  select  the 
best  one. 

2. 

Predict  what  is  likely  to  occur  if  the 
hydraulic  fluid  were  contaminated. 
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THE  USE  LEVEL  CAN  BE  FURTHER  DIVIDED  INTO  TWO  TYPES: 


1.  USE-UNAIDED  in  which  the  student  has  no  aids  except  his  own 
memory. 

2.  USE-AIDED  in  which  the  student  has  a  job  aid  for  performing  the  task. 

For  this  level,  the  nature  of  the  aid  depends  on  the  content  type: 

For  USE-AIDED  CONCEPTS  the  aid  should  consist  of  a  decision 
strategy,  including  each  critical  characteristic,  and  the  decision  to  be 
made  according  to  presence  or  absence  of  that  characteristic.  In 
simple  cases,  the  aid  may  only  include  a  list  of  characteristics;  the 
decision  strategy  is  then  implied. 

For  USE-AIDED  PROCEDURES  the  aid  would  be  a  list  of  steps  to  be 
performed. 

For  USE-AIDED  RULES  the  aid  would  be  at  least  a  statement  of  the 
formula  or  rule  to  be  applied,  and  could  include  guidelines  for  when 
and  how  to  apply  it. 

For  USE-AIDED  PRINCIPLES  the  aid  would  also  be  at  least  a 
statement  of  the  principle,  and  could  include  guidelines  for  when  and 
how  to  apply  it.  It  could  list  other  relevant  principles,  rules, 
procedures,  or  concepts  needed  to  apply  the  principle. 

EXAMPLES: 

USE-AIDED:  A  pilot's  preflight  checklist  is  a  USE-AIDED  procedure. 

The  pilot  does  not  have  to  remember  the  steps  or  their 
order  because  they  are  on  the  checklist.  The  pilot  does 
need  to  perform  the  steps  correctly. 

USE-UNAIDED:  "Field-strip  an  M-16  rifle." 

Here,  the  student  must  remember  the  steps  in  the  correct 
orcbr,  and  perform  them  correctly. 

In  summary:  The  REMEMBER  level  involves  "pure"  remembering, 

The  USE-UNAIDED  level  involves  remembering  what  is  to  be  used,  and 
then  using  it,  and 

The  USE-AIDED  level  involves  "pure"  using. 


TASK  LEVEL 


THE  ENTIRE  TASK/CONTENT  MAfRIX  IS  SHOWN  BELOW: 


FACT 

RECALL  OR 

RECOGNIZE 

NAMES. 

PARTS. 

OATES. 

PLACES.  ETC. 


CONTENT  TYPE 


CONCEPT 
REMEMBER 
CHARACTERIS¬ 
TICS.  OR 
CLASSIFY  OB¬ 
JECTS.  EVENTS 
OR  IDEAS  AC¬ 
CORDING  TO 
CHARACTERISTICS 


PROCEDURE 
SEQUENCE  OP 
STEPS  REMEM¬ 
BERED  OR  USED 
IN  A  SINGLE 
SITUATION  OR 
ON  A  SINGLE 
PIECE  OF 
EQUIPMENT 


RULE 

REMEMBER  OF 
USE  A  SEQUENCE 
OF  STEPS  WHICH 
APPLY  ACROSS 
SITUATIONS 
OR  ACROSS 
EQUIPMENTS 


PRINCIPLE 
REMEMBER  OR 
USE  A  GENERAL 
RULE  WHICH 
SELECTS  OTHER 
PRINCIPLES. 
RULES.  PROCE¬ 
DURES.  OR 
CONCEPTS.  AND 
APPLIES  THEM 
TO  SOLVE 
PROBLEMS 


REMEM8ER  -  RECALL  OR 
RECOGNI2E  FACTS.  CON¬ 
CEPT  DEFINITIONS.  STEPS 
OP  PROCEDURES  OR  RULES. 
STATEMENTS  OP  PRINCIPLE 


USE-UNAIDEO  *  TASKS  WHICH  REQUIRE 
CLASSIFYING.  PERFORMING  A  PROCEDURE. 
USING  A  RULE.  OR  APPLYING  A  PRINCIPLE 
WITH  NO  AIDS  EXCEPT  MEMORY. 


USE-AtDED  .  SAMI  AS  USC-UNAIOEO. 
EXCEPT  JOB  AIOS  ARE  AVAILABLE. 


NOTE:  (I)  Any  objective,  test  item,  or  piece  of  instruction  will  be 

classifiable  in  one  and  only  one  cell  of  the  matrix  above. 

(2)  This  matrix  is  used  throughout  the  TEE  guidelines. 


REVIEW? 

Task  level  and  content  type  are  ascertained  from  these  respective 
considerations: 

U  What  the  student  must  do;  that  is,  the  TASK  to  be  performed. 

2.  The  type  of  information  the  student  must  learn;  that  Is,  the 

instructional  CONTENT. 

Content  types  and  task  levels  are  described  again  in  the  reformatted 
task/content  matrix  below: 
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TASK  LEVEL 


CONTENT  TYPE 


FACT 


RECALL  OR  RG- 
COGNIZE  NAMES, 
PARTS,  DATES. 
PLACES.  VO¬ 
CABULARY  DEF- 
REMEMBER  INITIONS,  ETC. 


USE-UNAIDED 


.  USE-AIDED 


CONCEPT  PROCEDURE  RULE  PRINCIPLE 


REMEMBER  THE 
CHARACTERISTICS 
OF  EACH  CON¬ 
CEPT  AND  THE 
GUIDELINES  FOR 
CLASSIFICATION. 

REMEMBER  THE 
STEPS  OF  THE 
PROCEDURE. 

REMEMBER  THE 
FORMULA  OR 

THE  STEPS  OF 

THE  RULE. 

REMEMBER  THE 
RELATED  PROCE¬ 
DURES,  OR  CON¬ 
CEPTS  TO  BE 
SELECTED  FROM 
AND  APPLIED  OR 
THE  STATEMENT 

OF  THE  PRINCIPLE. 

CLASSIFY  OR 
CATEGORIZE 
OBJECTS.  EVENTS, 
IDEAS',  ACCORDING 
TO  THEIR  CHARAC¬ 
TERISTICS,  WITH 

NO  MEMORY  AID. 

APPLY  THE 

STEPS  OF  THE 
PROCEDURE  IN 

A  SINGLE  SIT¬ 
UATION  OR  ON 

A  SINGLE  PIECE 
OF  EQUIPMENT, 
WITH  NO  MEM¬ 
ORY  AID. 

APPLY  THE 
FORMULA  OR 
RULE  TO  A 
VARIETY  OF 
PROBLEMS  OR 
SITUATIONS. 
WITH  NO  MEM¬ 
ORY  AID. 

USE  THE  PRIN¬ 
CIPLE  TO  DES¬ 
CRIBE  WHY  OR 

HOW  THINGS 
HAPPEN  OR  WHAT 
WILL  HAPPEN, 

WITH  NO  MEMORY 
AID. 

GIVEN  CONCEPT 
CHARACTERIS¬ 
TICS  AND  GUIDE* 
LINES,  CATE¬ 
GORIZE  OBJECTS. 
EVENTS.  IDEAS, 
ACCORDING  TO 
CHARACTERIS¬ 
TICS. 

GIVEN  STEPS  OF 
THE  PROCEDURE. 
APPLY  THE  PRO¬ 
CEDURE  IN  A 
SINGLE  SIT¬ 
UATION.  OR 

ON  A  SINGLE 

PIECE  OF 
EQUIPMENT. 

GIVEN  THE 
FORMULA  OR 
RULE  STEPS. 
APPLY  THE 
FORMULA  OR 
RULE  TO  A 
VARIETY  OF 
PROBLEMS  OR 
SITUATIONS. 

GIVEN  A  STATE¬ 
MENT  OF  THE 
PRINCIPLE, 
DESCRIBE  WHY 

OR  HOW  THINGS 
HAPPEN  OR 

WHAT  WILL  HAP- 
FEN. 
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PROCEDURE  FOR  CLASSIFICATION 


Step  1.  Determine  the  TASK  LEVEL. 

a.  Decide  whether  the  student  is  to  REMEMBER  or  USE 
information. 

b.  If  the  student  is  to  USE  information,  decide  whether  the  task 
level  is  USE-AIDED  or  USE-UNAIDED. 

Step  2.  Determine  the  CONTENT  TYPE. 

a.  If  the  student  must  recall  or  recognize  names,  parts,  locations, 
functions,  dates,  places,  etc.,  then  the  content  type  is  FACT. 

b.  If  the  student  must  remember  characteristics  of  similar  objects, 
events,  or  ideas,  OR  if  the  student  must  sort  or  classify  objects, 
events,  or  ideas  according  to  characteristics,  then  the  content 
type  is  CONCEPT. 

c.  If  the  student  must  remember  a  sequence  of  steps  which  apply 
to  a  single  situation,  OR  if  the  student  must  apply  the  steps  to 
a  single  piece  of  equipment  or  a  single  situation,  then  the 
content  type  is  PROCEDURE. 

d.  If  the  student  must  remember  a  sequence  of  steps  and  decisions 
which  apply  in  a  variety  of  situations,  OR  if  the  student  must 
apply  the  sequence  across  a  variety  of  situations  or  types  of 
equipment,  then  the  content  type  is  RULE. 

e.  If  the  student  must  remember  a  general  rule  which  selects  or 
creates  other  principles,  rules,  procedures,  or  concepts;  remember 
such  related  content;  or  use  the  general  rule  to  solve  problems, 
then  the  content  type  is  PRINCIPLE. 


CLAS5IFICATON  PRACTICE 

In  this  section,  sample  objectives  and  sample  test  items  are  given  for  you  to 
classify  according  to  the  scheme  just  presented.  In  the  space  provided  you  can  give 
your  reasons  for  the  classification. 

Some  of  the  objectives  and  test  items  will  be  difficult  to  classify.  There  are  three 
reasons  for  this.  First,  many  of  them  are  not  "good”  ones;  they  are  written  in  such 
a  way  that  it  may  not  be  clear  what  behavior  is  required  or  what  content  is  to  be 
taught.  (Tliey  are,  however,  fairly  typical.)  Second,  all  of  them  are  taken  out  of 
context  and  may  deal  with  unfamiliar  topics.  Therefore,  ttey  arc  difficult  to 
classify  because  Information  about  the  job  is  not  provided.  Third,  some  examples 
were  chosen  deliberately  to  be  hard  to  classify,  so  that  classification  problems  could 
be  illustrated. 

For  these  reasons  you  should  not  expect  to  correctly  classify  all  of  these  objectives 
and  test  items  (or  any  others)  Immediately. 

When  you  finish  classifying  each  item  yourself,  turn  the  page  for  the  FEEDBACK  on 
that  item.  Look  at  the  reasons  for  the  item's  classification  and  compare  them  to 
your  own.  F-S 


OBJECTIVE:  Without  the  aid  of  references,  list  in  the  correct  order  the  steps 
for  shifting  from  a  lower  to  a  higher  gear  in  a  manua! 
transmission  car. 


Task  Level? 


Content  Type? 
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OBJECTIVE:  Without  the  aid  of  references,  list  in.  the  correct  order  the  steps 
for  shifting  from  a  lower  to  a  higher  gear  on  a  manual 
transmission  car. 


Task  Level?  REMEMBER _ 

Since  the  student  is  "listing”  a  series  of  steps,  rather  than  actually 
performing  them,  the  task  level  is  "Remember.” 


Content  Type?  PROCEDURE 

This  is  clearly  "Procedure"  content  because  there  is  a  sequence  of  steps 
(the  steps  involved  in  shifting  gears);  the  problem  to  be  solved  always 
presents  itself  in  the  same  way  (or  a  requirement  to  shift  to  a  higher 
gear);  and  there  is  only  one  way  to  upshift  correctly. 


OBJECTIVE:  Calculate  the  cost  of  gas  per  mile  to  the  nearest  tenth  of  a 

cent  given  (1)  miles  driven,  (2)  gallons  of  gasoline  used,  (3)  cost 
per  gallon  of  gas,  and  the  following  formula: 

Cost  of  gas  per  mile  =  C  =  ~ 

where  M  =  miles  driven,  to  nearest  mile 

G  =  gallons  of  gasoline,  to  nearest  tenth  gallon 
C  =  cost  of  gasoline  per  gallon 


Task  Level? 


Content  Type? 
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2. 


OBJECTIVE:  Calculate  the  cost  of  gas  per  mile  to  the  nearest  tenth  of  a 

cent  given  (1)  miles  driven,  (2)  gallons  of  gasoline  used,  (3)  cost 
per  gallon  of  gas,  and  the  following  formula: 

Cost  of  gas  per  mile  -  C  =  ~ 

where  M  =  miles  driven,  to  nearest  mile 

G  -  gallons  of  gasoline,  to  nearest  tenth  gallon 
C  =  cost  of  gasoline  per  gallon 


Task  Level?  USE-AIDED _ 

The  student  is  asked  to  perform  a  task  (calculate  cost  of  gas  per  mile). 
He  is  not  merely  remembering  something,  he  must  "use"  the  formula  to 
solve  for  the  unknown  value.  Since  the  formula  is  provided  to  the- 
student  the  task  is  "aided." 


Content  Type?  RULE _ 

The  verb  "calculate"  and  the  formula  provided  indicate  that  an  ordered 
series  of  steps  is  involved  in  solving  this  problem.  What  distinguishes 
this  from  a  procedure  is  that  these  steps  are  designed  to  work  for  any 
class  of  problems.  The  number  of  miles  driven,  gallons  of  gasoline 
used,  and  cost  per  gallon  of  gas  can  vary  widely,  but  the  formula  can 
still  be  used  in  all  cases. 
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OBJECTIVE:  Describe,  without  references,  the  critical  characteristics 

distinguishing  rockets  from  missiles,  according  to  the  course  text* 


Task  Level? 


Content  Type? 
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0B3ECTIVE:  Describe,  without  references,  the  critical  characteristics 

distinguishing  rockets  from  missiles,  according  to  the  course  text. 


Task  Level?  REMEMBER 

The  student  is  asked  to  describe  something  without  references,  i.e., 
from  memory,  so  the  task  level  is  "Remember." 


Content  Type?  CONCEPT 

The  term  "rocket"  is  a  Concept  name  referring  to  a  class  of  objects 
which  have  a  jet  engine  and  carry  either  liquid  or  solid  propellants 
which  make  the  engine  independent  of  oxygen  In  the  air. 
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OBJECTIVE:  Without  references,  label  the  three  basic  components  of  the  M-16 
rifle. 


Task  Level? 


Content  Type? 


OBJECTIVE:  Without  references,  label  the  three  basic  components  of  the  M-16 
rifle. 


Task  Level?  REMEMBER _ 

The  student  must  remember  the  basic  components  of  the  M-16  rifle.  It 
is  not  necessary  for  this  information  to  be  applied. 


Content  Type?  FACT _ 

The  content  is  a  Fact.  It  is  a  simple  association  between  symbols  * 
(component  names)  and  objects  (components  of  a  rifle). 
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5.  0B3ECTIVE:  Without  reference  to  TM  9-2350-258-10,  remove  and  replace  a 
defective  domeiight  lamp  in  the  M48A5  tank,  given  a 
screwdriver,  replacement  lamp,  and  clean  cloth. 


Task  Level? 


Content  Type? 
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OBJECTIVE;  Without  reference  to  7M  9-2350-258-10,  remove  and  replace  a 
defective  domelight  lamp  on  the  M48A5  tank,  given  a 
screwdriver,  replacement  lamp,  and  clean  cloth. 


Task  Level?  USE-UNAIDED _ 

The  task  level  is  "Use",  since  the  task  actually  involves  removing  and 
replacing  the  defective  lamp,  not  merely  recalling  how  to  do  it.  It  is 
"unaided”  because  the  student  has  no  job  aid  (in  this  case  the  TM)  to 
guide  him  in  performing  the  task.  Note  that  although  he  is  given  some 
items  (screwdriver,  lamp,  and  cloth)  which  are  necessary  for  task 
performance,  they  are  not  job  aids. 


Content  Type?  PROCEDURE _ 

The  content  is  a  Procedure.  It  is  a  set  of  sequential  operations  or 
steps  performed  on  a  single  object  (a  M4SA5  domelight)  in  a  specific 
situation  (domelight  is  defective).  The  task  can  be  performed  in  only 
one  way  (according  to  the  TM). 
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6.  OBJECTIVE:  Solve  for  total  power  in  a  DC  parallel  circuit. 


Task  Level? 


Content  Type? 


F-19 


Km lL«Tli/'*i’lL*llVlLkTtjA  W*‘*V"*;J*  JATJMAli  VM WVh Ifc'AAArt. 


OBJECTIVE:  Solve  for  total  power  in  a  DC  parallel  circuit. 


Task  Level?  USE-UNAIDED 

The  task  level  is  Use  because  the  student  must  solve  problems.  It  is 
not  aided  because  the  formula  for  total  power  is  not  given. 


Content  Type?  RULE _ 

The  word  solve  is  a  "keyword”  for  Rule.  Here  the  student  must 
remember  the  formula  for  total  power  and  then  use  it  to  solve 
problems. 


Test  Item;  Destroy  classified  documents  under  routine  conditions  given  the 
outline  in  OPNAVINST  5510.1  and  KAG-1. 


Task  Level? 


Content  Type? 


Test  Item: 


Destroy  classified  documents  under  routine  conditions  given  the 
outline  in  OPNAVINST  5510,1  ar.d  KAG-1. 


Task  Level?  USE- AIDED _ 

What  does  the  student  have  to  do?  He  must  "destroy"  classified 
documents.  However,  he  is  given  references  which  outline  the  steps 
that  must  be  performed  to  accomplish  this.  Therefore,  the  task  level 
is  use-aided. 


Content  Type?  PROCEDURE 

The  student  must  follow  a  specific  series  of  steps  for  destroying 
classified  documents  under  routine  conditions  -  one  situation.  This 
implies  that  there  are  other  specific  procedures  for  emergency 
conditions.  You  could  check  with  a  subject  matter  expert  to  make 
sure. 


8.  Test  Item:  State  the  radar  signal  characteristics  used  in  identifying  circular, 
sector,  conical  and  steady  scans  on  an  analysis  scope. 


Task  Level? 


Content  Type? 
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Test  Items  State  the  radar  signal  characteristics  used  in  identifying  circular, 
sector,  conical,  and  steady  scans  on  an  analysis  scope. 


Task  Level?  REMEMBER _ 

The  student  is  asked  to  state  (recall)  the  characteristics  of  four 
different  types  of  scans.  At  this  time  he  is  not  asked  to  actually 
categorize  a  signal  as  a  certain  type  of  scan.  Therefore,  the  task 
level  is  Remember. 


Content  Type?  CONCEPT _ _ 

The  way  this  test  item  is  written,  it  sounds  as  if  the  student  is 
remembering  characteristics  so  that  he  can  later  classify  signals. 


OBJECTIVE:  Recall  the  proper  sequence  of  steps  for  testing  a  capacitor  using 
a  standard  volt-ohm  meter. 


Task  Level? 


Content  Type? 


9. 


OBJECTIVE:  Recall  the  proper  sequence  of  steps  for  testing  a  capacitor  using 
a  standard  volt-ohm  meter. 


Task  Level?  REMEMBER _ 

Does  the  student  have  to  remember  something  or  actually  perform  the 
task?  In  this  case  he  must  only  recall  the  steps.  He  is  not  asked  to 
use  the  meter  while  testing  a  capacitor  to  see  if  anything  is  wrong 
with  it. 


Content  Type?  PROCEDURE _ 

It  seems  like  these  steps  would  be  the  same  regardless  of  the  type  of 
capacitor.  The  content  type  is  Procedure. 
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10*  Test  Item:  State  Ohm's  Law. 


Task  Level? 


Content  Type? 


F-27 


10.  Test  Item:  State  Ohm’s  Law. 


Task  Level?  REMEMBER _ 

State  is  a  keyword  for  the  Remember  level.  The  student  is  not  asked 
to  use  the  information  about  Ohm’s  Law  -  only  to  Remember  it. 


Content  Type?  RULE _ 

Ohm’s  Law  is  a  formula  (a  rule)  that  can  be  used  to  solve  for  an 
unknown  (I  =  E/R). 
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Test  Item:  Troubleshoot  electronic  circuits  from  PATRIOT  components  in 
which  no  current  is  flowing. 


Task  Level? 


Content  Type? 
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11.  Test  Item:  Troubleshoot  electronic  circuits  from  PATRIOT  components  in 
which  no  current  is  flowing. 


Task  Level?  USE-UNAIDED _ 

The  student  is  asked  to  perform  a  task.  It  is  not  just  recalling 
something;  he  must  remember  how  to  troubleshoot  a  circuit  and  then 
do  it.  No  aid  is  given. 


Content  Type?  PRINCIPLE _ 

The  student  must  use  his  knowledge  of  the  rules,  procedures,  and 
concepts  of  electronics  to  make  predictions  about  likely  causes  of 
trouble,  and  to  determine  what  tests  to  perform.  The  content  type  is 
Principle. 
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12.  OBJECTIVE:  Given  the  formula  for  capacitive  reactance  and  the  values  of 

frequency  and  capacitance  from  a  schematic,  calculate  capacitive 
reactance. 


Task  Level? 


Content  Type? 
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12.  0B3ECTIVE:  Given  the  formula  for  capacitive  reactance  and  the  values  of 

frequency  and  capacitance  from  a  schematic,  calculate  capacitive 
reactance. 


Task  Level?  USE-AIDED _ 

What  must  the  student  do?  He  must  "calculate'1  capacitive  reactance 
given  two  values  and  the  formula.  In  this  case  the  formula  is  the 
memory  aid.  The  task  is  use-aided. 


Content  Type?  RULE _ 

"Calculate"  is  another  keyword  for  rule.  Formulas  and  mathematical 
calculations  are  rules  that  can  be  applied  to  various  situations  and 
values. 
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13.  OBJECTIVE:  List  the  materials  which  are  separable  from  lube  oil  by  the 
Sharpies  oil  purifier. 


Task  Level? 


Content  Type? 


13.  OBJECTIVE:  List  the  materials  which  are  separable  from  lube  oil  by  the 
Sharpies  oil  purifier. 


Task  Level?  REMEMBER _ 

The  student  is  asked  to  state  (recall)  the  materials  which  can  or  cannot 
be  separated.  He  is  not  asked  to  do  it.  Therefore,  the  task  level  is 
Remember. 


Content  Type?  FACT _ 

There  are  only  a  few  materials  to  remember  and  this  objective  can 
best  be  taught  as  a.  Fact.  This  FACT  will  provide  information 
necessary  for  operating  the  Sharpies  purifier. 


r- 34 


TEST  ITEM  (in  a  course  for  engineers  who 


will  eventual!1 


Explain  how  hydraulic  systems  work  as  described  i 
in  Chapter  9  of  the  text. 


Task  Level? 


Content  Type? 
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design  equipment): 
class  lectures  and 


14.  TEST  ITEM  (in  a  course  for  engineers  who  will  eventually  design  equipment): 


Explain  how  hydraulic  systems  work  as  described  in  class  lectures  and 
in  Chapter  9  of  the  text. 


Task  Level?  REMEMBER _ 

The  student  is  asked  to  explain  from  memory  what  he  learned  about 
how  hydraulic  systems  work.  It  is  clearly  Remember. 


Content  Type?  PRINCIPLE _ 

This  test  Hem  asks  the  student  to  explain  how  hydraulic  systems  work. 
The  "hows'1  are  rules  that  could  be  used  in  conjunction  with  principles 
to  solve  problems  related  to  hydraulic  systems  or  in  their  design. 
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OBJECTIVE:  Sort  pictures  of  clot'ua  into  stacks  by  types,  given  labeled 
illustrations  of  identified  cloud  types. 


Task  Level? 


Content  Type? 


15.  OBJECTIVE:  Sort  pictures  of  clouds  into  stacks  by  types,  given  labeled 
illustrations  of  identified  cloud  types. 


Task  Level?  USE-AIDED 


This  objective  is  an  example  of  a  pure  "using"  task.  The  illustrations 
of  identified  clouds  are  the  aid.  There  is  no  need  to  remember  the 
characteristics  of  each  type  cloud;  they  are  given.  The  student  is 
asked  to  categorize  pictures  of  new  examples  of  clouds  based  on  the 
characteristics  in  the  labeled  clouds. 


Content  Type?  CONCEPT _ 

There  are  many  variations  in  cloud  formations.  In  order  to  label  clouds 
the  students  must  classify  new  examples  according  to  their 
characteristics. 
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TEST  ITEM:  Use  your  oscilloscope  to  calibrate  the  10XMX-2817V  test  prod. 


i 


Task  Level? 


Content  Type? 


TEST  ITEM:  Use  your  oscilloscope  to  calibrate  the  1QXMX-2817V  test  prod. 


Task  Level?  USE-UNAIDED _ 

The  student  must  perform  the  task.  He  must  "calibrate"  the  test  prod. 
He  is  not  told  how  to  do  it;  he  must  perform  the  job  from  memory. 

The  task  is  use-unaided.  The  oscilloscope  is  a  tool  necessary  to 
perform  the  task,  but  is  not  an  aid  because  it  does  not  tell  him  how  to 
do  the  task. 


Content  Type?  PROCEDURE _ 

When  you  are  calibrating  a  specific  single  piece  of  equipment  there  are 
usually  steps  that  you  must  perform  in  a  certain  order  each  time  you 
do  the  job.  This  is  probably  a  Procedure. 
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17.  OBJECTIVE:  Given  an  incoming  signal,  the  EW  operator  will  identify  it  as 

being  emitted  from  a  particular  type  radar  on  a  particular  type 
of  platform. 


Task  Level? 


Content  Type? 
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17.  OBJECTIVE:  Given  an  incoming  signal,  the  E\V  operator  will  identify  it  as 

being  emitted  from  a  particular  type  radar  on  a  particular  type 
of  platform. 


Task  Level?  USE-UNAIDED _ . 

The  student  will  analyze  the  incoming  signal  and  classify  it  according 
to  specific  characteristics.  The  task  level  is  use.  There  is  no  memory 
aid,  so  the  task  is  unaided. 


Content  Type?  CONCEPT _ 

From  an  analysis  of  the  job,  it  turns  out  that  there  are  many  incoming 
signals  which  can  be  classified  according  to  a  small  number  of  types  of 
radars  and  platforms. 
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IS.  OBJECTIVE:  Given  diagrams  that  graphically  represent  how  the  steam  cycle 
works,  use  them  to  discover  how  the  auxiliary  condensate  system 
works. 


Task  Level? 


Content  Type? 


» 
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OBJECTIVE:  Given  diagrams  that  graphically  represent  how  the  steam  cycle 
works,  use  them  to  discover  how  the  auxiliary  condensate  system 
works. 


Task  Level?  USE-AIDED _ 

What  must  the  student  do?  The  objective  requires  the  student  to  use 
information  about  the  stearn  cycle  to  explain  how  a  new  system  works. 
The  diagrams  are  given  to  "jog"  the  students’  memory  about  how  the 
basic  steam  cycle  works.  The  objective  is  use-aided. 


Content  Type?  PRINCIPLE 


Here,  principles  employ  other  rules  to  make  explanations,  analyses,  and 
predictions  about  how  things  work. 


SELF-TEST  ON  TASK  LEVEL  AND  CONTENT  TYPE 


Directions:  Below  each  objective  write  its  Task  Level  and  Content  Type.  Note 
that  these  are  not  necessarily  "good"  objectives,  but  their  task  levels 
and  content  types  should  be  clear. 


1.  Using  TM-9-1410-530-14  and  a  PATRIOT  launcher  station  prepared  for  reload 
in  a  simulated  tactical  field  environment,  perform  missile  reload. 

Task  Level:  _  Content  Type:  _ ; 


2.  Upon  receipt  of  a  warning  order  which  changes  the  mission  of  your  PATRIOT 
battalion  in  a  field  exercise,  determine  all  specified  and  implied  actions 
necessary  to  accomplish  the  mission. 

Task  Level:  _ _  Content  Type:  _ _ 


3.  Given  the  guidelines  for  determining  message  security  classification,  determine 
the  security  classification  (Top  Secret,  Secret,  Confidential,  or  Unclassified) 
for  outgoing  messages 

Task  Level:  _ _ Content  Type: _ 


4.  Given  a  picture  of  a  Vulcan  operator's  console,  state  the  function  of  each 
control  and  indicator  as  described  in  TM  9-2350-300-10. 

Task  Level:  _  Content  Type:  _ 


5.  Recall  in  writing  the  critical  characteristics  of  ten  types  of  Soviet  aircraft. 


Task  Level: 


Content  Type: 


6.  Given  the  formula  for  Ohm's  Law,  E=IR,  and  any  two  values  typical  of  the 
values  encountered  on  the  job,  solve  for  the  third  value  correct  to  two 
decimal  places. 

Task  Level:  _ _  Content  Type: _ 


7.  Given  an  ammunition  storage  locker  containing  four  different  kinds  of  Vulcan 
rounds  and  requests  for  certain  kinds  of  ammunition,  identify  the  appropriate 
rounds  to  fill  each  request. 

Task  Level: _  Content  Type: _ _ 


8.  State  the  general  guidance  for  applying  the  rules  of  war  and  the  tactical 

standard  operating  procedures  that  govern  the  conduct  of  an  IHAWK  battery 
as  contained  in  TM  430-7. 

Task  Level: _ Content  Type: _ 


9.  Given  a  globe  valve,  rags,  Prussian  blue,  gasket  material,  packing,  and  tools, 
disassemble  and  reassemble  the  globe  valve. 

Task  Level:  _  Content  Type:  _ 


10.  In  accordance  with  FM  44-1,  ACP  125,  and  TC  32-20,  take  the  prescribed 
actions  in  response  to  simulated  enemy  employment  of  electronic  warfare 
which  includes  locating  your  transmitters,  jamming,  and  imitative 
communications  deception.  There  is  no  time  to  look  at  the  reference 
documents  during  the  exercise. 

Task  Level:  _  Content  Type:  _ 


11.  State  the  method  for  selecting  the  correct  Vulcan  firing  mode  as  described  in 
TM 9-2350-300- 10.  Include  the  types  of  targets  and  conditions  under  which 
each  mode  is  used. 

Task  Level:  _  Content  Type: _ 


12.  Given  an  explanation  of  how  to  supply  military  troops  from  the  text, 

illustrations  to  include  Hitler's  attack  of  Russia,  the  Hattie  of  Midway,  the 
Battle  of  the  Bulge,  and  Sherman’s  march  through  Georgia,  and  a  modern 
tactical  scenario  you  have  not  seen  before,  discuss  how  the  types  of  troops  In 
the  scenario  should  be  supplied. 

Task  Level: _ Content  Type:  _ 


13.  List  the  steps  in  boresighting  the  2Q-MM  M163A1  cannon. 
Task  Level: _ Content  Type:  _ _ 
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KEY  TO  SELF-TEST  ON  TASK  LEVEL  AND  CONTENT  TYPE 


The  answers  are  given  below  followed  by  the  reasons  they  are  correct.  The  reasons 
are  based  on  the  definitions  of  the  task  levels  and  content  types. 


1.  USE-AIDED/PROCEDURE:  The  objective  is  performed  with  the  aid  of  the 
TM.  The  action  involves  a  sequence  of  steps  used  on  a  single  piece  of 
equipment. 


2. 


USE-UNAIDED  PRINCIPLE?  Nc  job  aids  are  specified  by  the  objective.  The 


action  involves  selecting  the  appropriate  principles,  rules,  and  procedures  from 
Army  doctrine  and  regulations,  to  be  applied  in  accomplishing  the  unit's 
mission. 


3.  USE-AIDED/CONCEPT:  The  "guidelines"  mentioned  are  a  job  aid.  The  action 
involves  classifying  events  according  to  the  critical  characteristics  of  the 
different  levels  of  classified  information. 


4.  REMEMBER/FACT:  The  objective  requires  information  to  be  recalled,  i.e., 
the  functions  of  parts.  These  functions  are  not  to  be  recalled  as 
characteristics  of  concepts,  steps  in  a  procedure  or  rule,  or  as  parts  of  a 
principle  and  are  therefore  facts. 

5.  REMEMBER/CONCEPT;  The  objective  requires  information  to  be  recalled. 

The  items  to  be  recalled  are  the  critical  characteristics  of  concepts. 

6.  USE-AIDED/RULE:  The  formula  given  in  the  objective  is  to  be  used  as  a  job 
aid.  The  application  of  Ohm's  law  entails  a  sequence  of  steps  which  apply 
across  a  number  of  situations. 


7.  USE-UNAIDED/CONCEPTj  No  job  aids  are  specified  by  the  objective.  The 
action  involves  classifying  objects  according  to  their  characteristics. 

8.  REMEMBER/PRINCIPLE:  The  objective  requires  the  recall  of  information. 

The  items  to  be  recalled  are  principles  (the  "general  guidance")  used  to  select 
other  rules  and  procedures  that  would  be  applied  in  solving  the  problem, 
which  in  this  case  would  be  the  performance  of  an  IHAWK  mission. 

9.  USE-UNAIDED/PROCEDURE:  No  job  aids  are  specified  by  the  objective. 

The  action  involves  using  a  sequence  of  steps  which  apply  to  a  single  piece  of 
equipment. 

10.  USE-UNAIDED/RULE:  The  subject  matter  must  be  understood  well  enough  to 
be  used  without  aids.  The  actions  involved  require  the  use  of  sequences  of 
steps  which  apply  across  different  situations. 

11.  REMEMBER/RULE:  The  objective  requires  the  recall  of  information.  The 
item  to  be  recalled  is  a  sequence  of  steps  which  apply  across  situations. 
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USE-AIDED/PRINCIPLE:  The  explanation  and  illustrations  provide  a  job  aid. 
Discussing  troop  supply  in  the  scenario  requires  the  use  of  information  not 
just  remembering  it.  The  action  required  involves  the  use  of  some  general 
rule  with  which  rules,  procedures,  and  concepts  on  the  supply  of  military 
troops  are  selected  and  applied.  (A  student  could  feel  that  the  objective 
describes  a  rule,  which  it  would  if  the  explanation  from  the  text  was  well 
defined  enough  to  prescribe  specific  courses  of  action  for  any  situation.  The 
fact  that  illustrations  are  given,  however,  suggests  that  the  method  is  not 
that  well  defined  and  is  therefore  a  principle.) 

REMEMBER/PROCEDURE:  The  objective  requires  the  recall  of  information. 
The  items  to  be  remembered  are  a  sequence  of  steps  which  are  used  on  a 
single  piece  of  equipment. 
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ADDITIONAL  PRACTICE  OB3ECTTVE5  ON  TASK  LEVEL 
AND  CONTENT  TYPE  WITH  FEEDBACK 


Use  the  objectives  below  or  additional  practice  on  task  level  and  content  type. 
FEEDBACK  classifying  each  objective  and  giving  reasons  for  the  classification 
follows.  No:-  again  that  these  are  not  necessarily  "good"  objectives,  and  some 
could  actually  be  classified  as  "goals"  or  "missions." 


1.  State  from  memory  the  primary  mission  of  the  Chaparral  Air  Defense  Guided 
Missile  System  as  described  in  FM  44-4. 


2.  Describe  the  method  used  by  personnel  in  a  Chaparral  radar  platoon  to  detect 
targets  at  or  near  maximum  range. 


3.  As  commander  of  a  Vulcan  Battery  in  a  field  exercise,  issue  a  warning  order 
to  the  platoons.  It  should  include  friendly  and  enemy  situations,  tentative 
primary  and  alternate  positions,  tentative  primary  and  alternate  routes,  and 
any  special  instructions  that  deviate  from  unit  SOP. 

4.  In  a  simulated  electronic  warfare  (E\V)  environment,  recognize  the  different 
type  of  external  jamming  encountered:  spot  jamming,  barrage  jamming, 
accidental  interference,  and  atmospheric. 


5.  Given  TM  9-2350-300-10  perform  before  operations  preventative  maintenance 
checks  and  services  (PMCS)  on  the  Vulcan  Carrier  (M163A1  System). 


6.  Describe  the  features  of  the  Vulcan,  Chaparral,  SGT  YORK  gun,  STINGER, 
REDEYE,  ROLAND,  HAWK,  1HAWK,  Nike  Hercules,  and  PATRIOT  systems  in 
order  to  be  able  to  distinguish  between  them  when  referenced  in  later 
training. 


7.  State  the  actions  a  gunner  must  take  Immediately  In  the  event  of  a  potential 
cook-off,  hang-fire,  or  stoppage  when  firing  the-  Vulcan  cannon. 


S.  Using  target  drones,  towed  targets,  or  BATS  representing  enemy  aircraft,  the 
Chaparral  squad  will  detect,  identify,  acquire,  engage,  and  kill  targets,  taking 
different  types  of  courses  at  varying  speeds  and  altitude.*. 


9.  Given  a  coordinate  square,  a  map  with  a  legend  identifying,  map  symbols,  and 
a  programmed  text  on  basic  map  rending  methods,  find  the  distance  between 
two  points  specified  by  symbols  in  designated  grid  squares. 
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10.  Given  a  manual  containing  guidelines  for  conducting  an  evaluation  of  an  Army 
training  course,  evaluate  the  training. 


11.  Using  a  chart  showing  the  characteristics  of  a  properly  maintained  spark  plug, 
sort  spark  plugs  into  two  groups:  acceptable  and  defective. 


12.  Given  proper  tools  in  a  motor  pool  shop,  replace  a  front  wheel  bearing  on  a 
1/4-ton  truck  in  less  than  10  minutes. 


13.  List  the  points  a  battery  air  defense  commander  must  consider  in  defending 
the  battery  against  a  ground  attack  assuming  that  early  warning  outposts 
detect  the  enemy  in  sufficient  time  to  allow  deployment  of  the  battery  to 
supplementary  defensive  positions.  »e  must  insure  that: 

•  The  battery  operations  center  is  notified  of  impending  attack 
without  compromising  any  information  of  use  to  the  enemy  force. 

•  A  reaction  force  is  assembled  and  tactically  employed. 

•  Battery  assets  available  for  defeating  the  enemy  force  are 
employed  in  a  manner  appropriate  with  tenain. 

•  Weapon  positions  have  engaged  the  enemy  at  maximum  range. 

•  Mixed  antitank/antipersonnel  forces  are  deployed  to  locations 
external  to  the  battery  position  area  if  appropriate. 
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FEEDBACK 


REMEMBER/FACT:  The  information  must  be  recalled  from  memory.  The 
mission  is  to  be  simply  stated  as  a  fact.  It  is  not  a  concept  characteristic 
or  a  step  in  a  procedure,  rule,  or  principle. 

REMEMBER/RULE:  The  information  is  to  be  recalled  rather  than  applied. 

The  information  is  a  sequence  of  steps  that  applies  across  situations,  e.g., 
different  terrain,  different  weather  conditions,  aircraft  at  different  altitudes, 
etc. 

USE-UNAIDED/PRINCIPLE:  No  job  aid  is  specified  in  this  performance 
objective,  and  none  would  typically  be  used  in  the  field.  The  action  involves 
the  use  of  principles  to  select  and  apply  other  principles,  rules,  procedures, 
and  concepts  to  determine  and  describe:  the  friendly  and  enemy  situations, 
tentative  positions  and  routes,  and  special  instructions;  and  to  transmit  the 
order  to  the  platoons. 

USE-UNAIDED/CONCEPT:  No  job  aid  is  specified  for  this  performance 
objective.  The  action  involves  classifying  events,  i.e.,  the  different  types  of 
jamming  received,  according  to  their  characteristics. 

USE-AIDED/PROCEDURE:  The  TM  is  specified  as  a  job  aid  for  use  in 
performing  the  objective.  The  action  involves  a  sequence  of  steps  applied  to 
a  single  piece  of  equipment. 

REMEMBER/CONCEPT:  The  information  is  to  be  recalled  rather  than 
applied.  The  action  involves  describing  the  characteristics  of  the  different . 
systems  in  order  for  the  student  to  be  abl:  to  classify  them  properly  in  later 
training. 

REMEMBER/PROCEDURE:  The  information  must  be  recalled  rather  than 
applied.  The  action  involves  stating  three  sequences  of  steps  which  apply  to 
a  single  piece  of  equipment. 

USE-UNAIDED/RULE:  No  job  aids  are  specified  by  the  performance  objective 
and  the  actions  called  for  leave  little  If  any  time  to  refer  to  a  job  aid.  The 
action  involves  performing  a  sequence  of  stops  which  apply  across  situations, 
the  most  important  of  which  are  specified. 

USE-AIDE*  The  map  legend  and  text  are  job  aids  for  this 

performance  objective.  The  action  Involves  performing  a  sequence  of  steps 
that  apply  across  situations,  i.e.,  different  symbols  in  different  squares. 

USE~A1DED/PRINCIPLE:  The  manual  is  a  job  aid;  the  objective  requires  a 
performance.  The  action  requires  use  of  principles  ("guidelines")  to  select  the 
appropriate  principles,  rules,  procedures,  and  concepts  of  evaluation  and  apply 
them  to  the  course. 

USE-AIOEQ/CONOEPT:  The  chart  is  a  jot*  aid  for  this  performance  objective. 
The  action  Involves  identifying  two  classes  of  spark  plugs  according  to  their 
cr i  t i cal  charar  tori sti c$. 


USE-UNAIDED/PROCEDURE:  No  job  aid  is  listed  in  this  performance 
objective.  The  action  involves  a  sequence  of  steps  used  on  a  single  piece  of 
equipment. 

REMEMBER/PRINICIPLE:  The  information  is  to  be  recalled  rather  than 
applied.  The  points  to  be  recalled  are  the  important  principles,  rules,  and 
procedures  which  would  be  selected  and  applied  in  solving  the  problem  of 
defending  an  air  defense  battery  against  ground  attack. 
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Appendix  G 


GUIDELINES  FOR  DETERMING  THE  SERIOUSNESS  OF 
TRAINING  DEFICIENCIES 

("2"  AND  "3W  RATINGS  ON  MASTER  LIST  QUESTIONS) 
(Used  in  conjunction  with  TEE  Task  E3) 


Note: 

a) 

Ratings  of  "l”  always  indicate  that  the  training  or  test  is  adequate 
on  the  point  in  question. 

(2) 

When  ratings  are  indicated  below  as  serious,  it  means  that  the  test 
item,  test,  presentation  component,  LO,  TLO,  or  LESSON  to  which 
they  apply  is  probably  totally  inadequate. 

(3) 

The  analyst  should  always  be  concerned  with  whether  data  co!lectorsT 
ratings  are  correct  and  should  not  necessarily  accept  the  judgments 
given  in  this  Appendix  without  carefully  weighing  all  factors  involved 
in  the  training  situation  under  evaluation  which  apply  to  the  Master 
List  questions  under  consideration. 

General  Considerations 


The  difficulty  of  a  given  task  is  often  an  important  factor  in  ascertaining  the 
seriousness  of  ratings  on  Master  List  questions.  The  difficulty  is  a  result  of  the 
combined  effect  of  two  factors: 

•  How  hard  the  task  is  to  perform  (or  remember)  in  terms  of  its  length, 
complexity,  or  requirements  for  fine  motor  manipulations. 

©  The  aptitudes  and  abilities  of  the  students  who  learn  the  task. 

To  illustrate  the  relationship  between  these  two  factors,  note  that  some  tasks 
which  are  difficult  for  average  students  may  be  relatively  easy  for  very  bright  students. 
Some  tasks  are  so  difficult  that  even  the  best  students  will  consider  them  difficult. 
Some  are  so  easy  that  even  the  poorest  students  will  have  no  trouble  with  them. 

A  good  test  to  determine  difficulty  (which  may  require  a  subject  matter  experts 
judgment)  is  to  ask:  Given  only  a  statement  of  how  to  perform  the  task  or  of 
information  to  be  remembered  and  a  short  period  of  time  to  study  it,  how  many  students 
would  be  able  to  perform  the  tasks  or  state  the  information  successfully? 

•  If  less  than  1/2  of  the  students  could  do  so,  the  task  is  hard. 

•  If  1/2  of  the  students  or  more  could  do  so,  the  task  is  easy. 
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Testing  Section 


In  this  section  a  "serious"  rating  means  one  of  three  things.  Either: 

o  A  substantial  number  of  students  (perhaps  20%  or  more)  who  should  pass 
a  test  item  or  the  test  (because  they  have  mastered  the  subject  matter) 
will  fail  it  (due  to  the  testing  deficiency  noted); 

o  A  substantial  number  of  students  who  should  fail  a  test  item  or  the  test 
(because  they  have  not  mastered  the  subject  matter)  will  pass  it  (due  to 
the  testing  deficiency  noted);  or 

o  A  combination  of  the  two  problems  above  will  cause  a  substantial  number 
of  students  to  either  pass  or  fail  who  should  not  pass  or  fail,  respectively. 


1.-4. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

SERIOUS 

HARD 

2 

SERIOUS 

3 

SERIOUS 

IF  RATING  IS: 

THEN  DEFICIENCY  IS: 

2 

MINOR 

3 

SERIOUS 

6.-7.  Examine  the  CONDITION  (6)  or  STANDARDS  (7)  which  arc  not  the  same  and 
make  a  judgment  on  the  seriousness  of  the  difference: 


IF  RATING  IS: 

THEN  DEFICIENCY: 

2 

May  or  may  not  he  serious 

3 

Is  probably  serious 
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8.-10. 


A  "3"  rating  is  serious  for  the  item. 


11.-12. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

SERIOUS 

HAUn 

2 

SERIOUS 

3 

SERIOUS 

13. 


IF  RATING  IS: 

THEN  DEFICIENCY  IS: 

2 

MINOR 

3 

SERIOUS 

14.  A  ',3,,  rating  is  minor  for  one  item;  however,  several  ”3”  ratings  on  different 
test  items  can  be  serious  for  the  test  as  a  whole. 

15. -17. 


IF  RATING  IS: 

THEN  DEFICIENCY  15: 

2 

MINOR 

3 

SERIOUS 

18. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY  FOR  AN 
INSTRUCTOR 

TO  REMEMBER 

2 

OK 

3 

MINOR 

HARD  FOR  AN 
INSTRUCTOR 

TO  REMEMBER 

2 

MINOR 

3 

SERIOUS 
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A  "S"  rating  is  serious. 


-20. 


-22.  A  ,?2n  or  ”3”  rating  is  minor. 


Examine  the  test  instructions  with  which  there  are  problems: 


IF  RATING  IS: 

THEN  DEFICIENCY  IS: 

2 

Is  probably  minor 

3 

May  or  may  not  be  serious 

IF  PERFORMING  THE 
TASKS  TOGETHER  IS: 

AND  THE 
RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR 

HARD 

2 

MINOR 

3 

SERIOUS 

IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR 

HARD 

2 

MINOR 

3 

SERIOUS 

IF  RATING  IS: 

THEN  DEFICIENCY  IS: 

2 

MINOR 

3 

SERIOUS 
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Presentation  Section 


As  a  general  rule  in  this  section,  do  not  consider  any  deficiency  serious  unless 
you  feel  that  it  Is  bad  enough  to  cause  many  students  (perhaps  20%  or  more)  to  fail 
the  final  test  for  the  task.  Judgments  below,  that  require  specific  consideration  of 
the  particular  deficiency  you  are  dealing  with,  are  noted  by  asterisks  (*)  and  are  • 
sometimes  given  further  consideration  below  the  tables  in  which  they  appear. 
Occasionally,  additional  criteria  are  given  there,  such  as  directions  to  consider  the 
effects  of  related  presentation  components. 


27.  Attention  getting/motivational  devices  are  required  when: 

•  The  student  would  consider  the  task  evaluated  long  and  boring  or 
difficult  to  learn,  and 

•  The  students  have  so  little  motivation  that  the  satisfaction  of  passing 
the  course  or  the  disappointment  of  failing  would  not  provide  enough 
motivation  to  overcome  the  problem  in  the  point  above. 

If  and  only  if  motivational  devices  are  required,  a  M3**  rating  is  serious.* 


28.  A  M3M  rating  is  serious.* 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

OK 

3 

MINOR 

HARD 

2 

SERIOUS 

3 

SERIOUS 

30.  A  ,,3,,  rating  is  serious  f  **  the  task  to  which  it  applies,  but  only  if  there  is  some 
other  evidence  that  the  students  do  not  have  the  prerequisites. 


31.  A  uy*  rating  is  minor  unless  you  have  evidence  that  the  lesson  was  so  disorganized 
that  it  was  unclear  what  terminal  performance  the  students  were  supposed  to  do 
or  learn  during  much  of  the  presentation. 

» 
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32. 


IF  TASK  IS: 

AND  RATING  IS: 

— 

THEF  DEFICIENCY  IS: 

EASY 

2 

OK 

3 

MINOR 

HARD 

2 

MINOR* 

3 

SERIOUS* 

33.  Look  at  the  specific  deficiency  noted  on  the  data  collection  worksheets.  Ask 
yourself  if  the  deficiency  would  have  caused  many  students  to  fail  the  final  test 
for  the  task  considering  the  effects  of  other  presentation  components  (i.e.,  did 
STATEMENT  HELP,  EXAMPLES,  or  PRACTICE  and  FEEDBACK  make  up  for  the 
deficiency  in  the  STATEMENT?).  If  the  deficiency  is  not  corrected  by  other 
components,  a  ,,2,,  rating  could  be  serious,  and  a  "3"  rating  is  probably  serious. 
Both  of  these  judgments  depend  on  the  probable  effect  of  what  is  missing  from 
the  STATEMENT  on  student  performance.  If  the  deficiency  is  corrected,  it  is 
minor. 


34.  A  "3"  rating  is  serious.* 


35. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

OK 

3 

MINOR 

HARD 

2 

MINOR* 

3 

SERIOUS* 

*Look  at  the  specific  deficiency  noted  on  the  data  collection  worksheets.  Ask 
yourself  if  that  deficiency  would  have  caused  many  students  to  fail  the  final 
test  for  that  task  considering  the  effects  of  other  presentation  components  (i.e., 
did  PRACTICE  and  FEEDBACK  or  EXAMPLES  make  up  for  the  deficiency  in 
tlie  STATEMENT  HELP?),  If  the  deficiency  is  not  corrected  it  could  be  serious. 
If  it  Is  corrected  it  is  minor. 
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36. 


IF  THE  JPA  15: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY  TO  USE 

3 

MINOR 

HARD  TO  USE 

3 

SERIOUS* 

37.  Lock  at  the  specific  deficiency  noted  on  the  data  collection  worksheets.  Ask 
yourself  if  the  deficiency  would  have  caused  many  students  to  fail  the  final  test 
for  the  task  considering  the  effect  of  other  presentation  components  (i.e.,  did 
the  STATEMENT,  STATEMENT  HELP,  EXAMPLE  HELP,  or  PRACTICE  and 
FEEDBACK  make  up  for  the  deficiency  in  the  EXAMPLE  or  MON-EXAMPLE?). 
If  the  deficiency  is  not  corrected  by  other  components,  a  ”3M  rating  is  probably 
serious.  That  judgment  depends  on  the  probable  effect  of  the  deficiency  on 
student  performance.  If  the  deficiency  is  corrected,  it  is  minor. 


38. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR* 

3 

MINOR* 

HARD 

2 

MINOR* 

3 

SERIOUS* 

*As  in  33,  34,  and  37,  consider  the  effect  of  other  components  such  as  PRACTICE 
and  FEEDBACK. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

3 

MINOR 

HARD 

3 

MINOR* 

♦This  deficiency  would  be  serious  only  with  very  hard  concepts  and  when  other 
factors  such  as  PRACTICE  and  FEEDBACK  do  not  make  up  for  it. 
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40.-42 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

3 

MINOR 

HARD 

3 

SERIOUS* 

*Again  consider  the  effect  of  other  components. 

43.-44. 


■^Consider  the  effects  of  other  components. 

45.  Memory  aids  are  required  only  when: 

•  The  content  to  be  memorized  is  long  or  complicated,  and/or 

•  When  the  information  is  very  new  to  the  students,  such  that  it 
cannot  be  easily  related  to  existing  knowledge. 

If  memory  aids  arc  required,  a  rating  Is  serious  only  when  many  students 
would  still  be  likely  to  fail  the  test  after  the  PRACTICE  and  FEEDBACK  which 
is  given. 


i, 


IF  TASK  IS 


AND  RATING  IS 


THEN  DEFICIENCY  IS: 


EASY 

3 

MINOR 

3 

SERIOUS* 

■49. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

3 

MINOR 

3 

SERIOUS* 

IF  PERFORMING  THE 
TASKS  TOGETHER  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

OK 

3 

MINOR* 

2 

MINOR* 

3 

SERIOUS* 

*52. 


IF  TASK  IS: 

AND  RATING  ISf 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR* 

2 

MINOR* 

3 

SERIOUS* 
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53, 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR* 

HARD 

2 

MINOR 

3 

SERIOUS 

IF  THE  TASK 
TESTED  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

3 

MINOR* 

HARD 

3 

SERIOUS* 

IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS; 

EASY 

2 

MINOR 

3 

MINOR* 

HARD 

2 

MINOR* 

3 

SERIOUS* 

56,-57.  Examine  the  CONDITIONS  (55)  or  STANDARDS  (56)  which  arc  not  the 
same  and  ma!:e  a  judgment  on  the  seriousness  of  that  difference; 


IF  TASK  IS: 

THEN  DEFICIENCY: 

EASY 

May  or  may  not  he  serious 

HARD 

Is  probably  senous 
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58.-61. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR* 

3 

SERIOUS* 

HARD 

2 

MINOR* 

3 

SERIOUS 

IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR* 

3 

SERIOUS 

HARD 

2 

MINOR* 

3 

SERIOUS 

IF  TASK  IS; 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

OK 

3 

SEE  NOTE 

HARD 

2 

MINOR* 

3 

SEE  NOTE 

NOTE:  Return  to  question  32  and  change  your  ratings  and  the  determination  of 
the  seriousness  of  deficiencies  for  the  task  to  reflect  the  fact  that  the 
FEEDBACK  component  is  missing,  if  that  is  not  already  the  case  there. 
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64.  This  question  applies  only  when  there  is  a  "l"  rating  on  question  63. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

SERIOUS* 

HARD 

2 

MINOR* 

3 

SERIOUS 

65.  With  respect  to  crew/team  (but  not  individual)  performance  scores,  a  ,,3"  rating 
is  serious. 


66.  Examine  the  TEAM  PRACTICE  CONDITIONS  which  arc  not  the  same  as  real 
world  CONDITIONS  and  make  a  judgment  as  to  the  seriousness  of  the  difference: 


— 

IF  RATING  IS: 

THEN  DEFICIENCY  IS: 

2 

May  or  may  not  be  serious 

3 

Is  probably  serious 

IF  THE  TEAM  FUNCTION: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

Involves  little  more  than 
performance  of  individual 
tasksjat_  the  same  time. 

2 

OK 

3 

MINOR 

Requires  a  fair  amount  of 
corn  m  uni  cat  ion  and 
coordination  among  team 
members  in  order  to 
accomplish  lr. 

2 

MINOR* 

3 

SERIOUS* 

6$,  This  question  applies  only  whan  then*  is  a  MIH  rating  e;i  question  67. 


C!3 


IF  THE  TEAM  FUNCTION: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

Involves  little  more  than 
performance  of  individual 
tasks  at  the  same  time. 

2 

MINOR 

3 

MINOR* 

Requires  a  fair  amount  of 
communication  and 
coordination  among  team 
members  in  order  to 
accomplish  it. 

2 

MINOR* 

3 

SERIOUS 

69.-70. 


IF  TASK  IS? 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR 

HARD 

2 

MINOR 

3 

SERIOUS* 

IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR* 

HARD 

2 

MINOR* 

3 

SERIOUS* 

IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

OK 

3 

MINOR- 

HARD 

2 

MINOR 

3 

MINOR* 
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73, 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

Easy  to  understand 
without  visuals 

2 

OK 

3 

MINOR* 

Hard  to  understand 
without  visuals 

2 

MINOR* 

3 

SERIOUS 

IF  THE  DEFICIENCY 
ON  QUESTION  73  IS: 

AND  THE  RATING  ON 
THIS  QUESTION  IS: 

THEN  DEFICIENCY  15: 

OK  (or  a  "l"  rating) 

2 

MINOR 

3 

MINOR* 

MINOR 

2 

SERIOUS* 

3 

SERIOUS 

SERIOUS 

2 

SERIOUS 

3 

SERIOUS 

75.-76. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

3 

MINOR* 

HARD 

3 

SERIOUS 

77.  A  M3"  rating  on  this  question  should  have  been  covered  elsewhere;  however,  it 
is  useful  for  purposes  of  revising  the  Instruction.  You  should  compare  ratings 
for  any  deficiencies  noted  here: 
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IF  DESCRIPTION  IS: 

SEE  RATINGS  ON  QUESTION  its 

NO  REAL  PRACTICE 

32,  44-66 

INCOMPLETE  INFORMATION 

34,  35,  37-43 

NO  FEEDBACK 

62,  63,  65,  66 

NO  HANDS-ON  PRACTICE 

32,  46-66 

A  ,f3"  rating  is  serious. 


IF  RATING  IS 


2 

3 


THEN  DEFICIENCY  IS: 


MINOR 

MINOR* 


Examine  any  unrealistic  demands.  A  "3"  rating  may  be  serious  if  it  adversely 
affects  the  student’s  ability  to  learn  in  the  course 


IF  RATING  IS 


THEN  DEFICIENCY  IS 


2 

MINOR* 

3 

SERIOUS 

IF  RATING  IS 


2 


3 


THEN  DEFICIENCY  IS: 


MINOR 


MINOR* 


IF  RATING  IS: 

THEN  DEFICIENCY  LS: 

2 

MINOR* 

3 

SERIOUS 

84.  A  ,,3”  rating  will  be  minor  unless  the  additional  content  seriously  confuses 

the  students  on  the  course  content. 


85.-86. 


A  '’2”  rating  is  minor. 

A  ,,3M  rating  is  serious.* 


87.-88. 


A  ^  rating  is  minor. 
A  "3"  rating  is  serious. 


89. 


A  "3"  rating  is  minor.* 


90.  A  "2"  rating  is  minor. 

A  "3M  ratin  is  serious.* 


91. 


IF  TASK  IS: 

AND  RATING  IS: 

THEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR 

HARD 

2 

MINOR 

3 

SERIOUS* 

*Consider  the  effect  of  other  components  such  as  PRACTICE  and  FEEDBACK. 


92.  A  l,2"  rating  is  minor. 

A  "3"  rating  is  serious. 


93.  Examine  the  training  device/cquipment  malfunction(s)  that  occurred  for 

the  task. 


IF  TASK  IS: 

AND  RATING  IS: 

‘I'HEN  DEFICIENCY  IS: 

EASY 

2 

MINOR 

3 

MINOR* 

HARD 

2 

MINOR* 

3 

SERIOUS* 

94.  Each  response  must  be  examined  and  judged  on  its  own  characteristics. 

C-17 


Appendix  H 

WORKSHEETS  FOR  CONDUCTING  THE  TEE 


Date: _ 

Evaluator: _ 

WORKSHEET  Ah  BACKGROUND  ON  THE  PROJECT 


Course: _ 

Purpose  of  Course: 


Primary  Audience:  _ 

Secondary  Audience:  _ 

Expected  Entry  Skills/Training: 


Purpose  of  TEE: _ 

User  of  TEE: _ _ 

User  Contact: _ 

Training  Dates: _ Location: 

Testing  Dates:  _ Location: 

Training  Contact: _ 

Limitations  on  Observations: _ _ 


Resource  Material: 

Exist? 

Source: 

Date 

Obtained: 

Date 

Revised: 

Task  Documentation  (from 
Task  Analyses) 

Training  Concept  Plan 

Lesson  Plan 

Practical  Exercises 

Soldier’s  Manual 

Course  Handouts 

Test  Instruments 

Commander's  Manual 

S.Q.T. 

! 

Training/Testing  Schedule 

Operator's  Mamnl 

Field  Manual 
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Worksheet  A1  (continued) 

Target  Date  for  TEE  completion: _ * _  Contingencies: 


Previous  TEE  on  course?  Yes  □ 

No  □ 


Report  available?  Yes  □ 

No  □ 


Comparative  Study  indicated?  YesD  No  □  Notes: 


Training  Development  Organization: _ 

Can  they  be  contacted?  Yes  □  No  □  Notes:  _ 

Delivery  Organization: _ _ _ 

How  will  trainees  be  selected? _ _ _ 

Entry  test  to  be  given?  Yes  □  No  □  Notes: _ 

Pretest  to  be  given?  Yes  □  No  □  Notes: 

How  will  instructors  be  selected? _ _ 

Qualifications  of  Instructors: 

Instructor/Trainee  Ratio: _ Notes: _ _ 

Instructor-dependence:  High  □  Medium  □  Low  □ 

Complexity  of  tasks:  High  □  Medium  □  Low  □ 

Ease  of  training  observation:  Difficult  □  Medium  □  Easy  □ 

Ease  of  testing  observation:  Difficult  D  Medium  □  Easy  □ 

Mission  criticality:  Very  high  □  High  □  Medium  □  Low  □ 

Other  Notes: _ _ _ _ _ 
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Evaluator 
Sheet  . 


WORKSHEET  A3:  SELECT  TASKS  FOR  EVALUATION 
Course;  _ 


Task 
ID  fi 

Known? 

Problems? 

Critical? 

Unique? 

Use? 

Other? 

Eliminate? 

Date: _ 

Evaluator:  _ 

Sheet  //: _ 

WORKSHEET  B2:  PRODUCT  EVALUATION  -  EVALUATE  TEST  MATERIALS 

Course: _ _ 

Test: _ - 

Follow  the  directions  for  Task  B2  in  the  User’s  Guide. 


Lesson  # 
and/or 
Title 

Task/ 
TLO  # 

LO  # 

Item  # 

TEE 

Question 

Rating 

(Circle) 

Description 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

I  2  3 

1  2  3 

l  2  3 

l  2  3 

1  2  3 

1  2  3 

l  2  3 
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Date: _ 

Evaluator: _ 

Sheet  _ 

WORKSHEET  B3:  PRODUCT  EVALUATION  -  EVALUATE  PRESENTATION 


Course:  _ 

Follow  the  directions  for  Task'B2  in  the  User's  Guide. 


Lesson  # 
and/or 
Title 

Task/ 
TLO  # 

LO  # 

S/E/P/F* 

TEE 

Question 

Rating 

(Circie) 

Description 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

1  2  3 

I  2  3 

1  2  3 

•Statement,  Examples,  Practice,  or  Feedback  H-6 


WORKSHEET  CU:  OBSERVATION  OF  CLASSROOM  INSTRUCTION 

Date: _ 

Data  Collector: _ 

Group/MOS  #% _ * _ Course:  _ 

LESSON  Hi _ LESSON  Title:  _ 

Applicable  TLO/LO//(s): _ 

Training  Location/Environment: _ _ _ ' 

Starting  Time:  _ _ Instructor  ID#(s):  _________ 

If  any  of  the  questions  below  are  not  applicable  to  the  training  event  you  are  observing, 
write  N/A  in  the  description  column. 


Make  one  rating  for  each  question.  If  several  things  could  be  rated  for  a  question, 
take  them  all  into  account  in  making  one  rating  for  the  event. 


TEC  QUESTION 

RATING  fOtlo  the  *~«ber) 

DCSCRIPnON 

27,  Aro  motivation*!  tectaiquM  tmpt  i? 

1  *  Y*»  (PI 04**  docrib*  ts«mj 

1  »  No 

21.  h  tfca  trolnt*  attltodo  pwltJva? 

1  *  Pc*  it  tv* 

2  •  Mi(fer*ftt 

S  ,  ItottU*  w  (tintraiad 

53.  i«  m*«t«ry  ol  pmequUito 

v*r*!»*d  prk*  to  Imtrvciloo? 

1  *  Ym 

2  «  No 

)l.  Aft  OS3CCTJVES  pt*%***4  to  ib* 

»Vid*ftt? 

1  *  Yri 
)  .  No 

•IK  Af*  STATEMENTS  ft*  CONCEH1S. 
rtoccoutw.  Of  nuus  *4r*»»*? 
(S*t  fob  aid  CTttofuJ 

|  *  C*M‘?!***i/  *d*(|U4t* 

)  *  Som*  or  *U  lvalue*,  emittod 

>*.  tee*  STATEMENT  HELP 

1  *  Mvtp  a#*vidr»  wllt‘ivn< 
*%pUft4ti*n. 

#  *  Hrlf  |Mt((Kiw| 

tiptanativ*. 

)  -  M*i#  t*  CtvdgM^, 

K.  Do#*  tr«m**e  m 

t So  «o*  of  rrqurffd  jp*b 
ait*’ 

io  y« 

1  *  No 

•ML  K  EXAMPLE  HUP  •***»*? 

t  *  Htt,  |v*r*A*  wlfiifHUl 
*p«^Un^k<A. 

1  *  ISrU»  jtm  mvmIIkW 

)  ,  lie!,  i*  iwl^. 

•  »  TM  only 
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WORKSHEET  CIA  (continued) 


Date: 


LESSON  Title: 


TEE  QUESTION 

RATING  (Circle  the  number) 

•39.  Are  EXAMPLES  tequenced  from  eaty 
to  hard?  (CONCEPTS  only) 

1  .  Yw 

3  *  No 

DESCRIPTION 


••0.  Are  there  enough  EXAMPLES?  (See  I  .  Yet 
Job  aid  lor  criteria.)  3  ■  No 


•it.  Are  NON-EXAMPLES  included? 
(CONCEPTS  only) 


•03.  Are  step*  in  a  DEMONSTRATION 
the  appropriate  lise?  (See  Job  aidJ 


R.  Are  ta*W>  and  tmk  ttrpt 

DEMONSTRATED  m  the  tame 
•erjuenre  m  they  are  performed  in 
the  real  world? 


*3.  Are  memory  aid*  gaed?  (PRACTICE 
REMEM&ER1NG  only) 


43,  N  PfEDAACK  provided  for 

pr  ac  nee? 


•«*.  |\  PLfrtftACK  HELP  a*Vf<Juatr'* 


t%  Ik  thr  I'tt'hrW'at  of  Wfwi*« 

tr  tpfkort  WktwUi  «4e*|uate?  fSw 
job  aid  for  tritrria,  Mate*  *****  an 
pohreO 


?).  h  Vh,'  warding  »l  WtMMk  «r  tfoLa* 
r«»  tor  the  rtudwitk  M 
wr<derM*i4? 


#2*  h  l*>*  «M!ie  ?a#S  park—**  i<*at  w 
the  e»kf  te  l*»M  *«*? 


T€t  .W,,r 


I  .  Ye* 
3  =  No 


*2.  Do  DEMONSTRATIONS  thaw  haw  to  I  *  Yes 
cor  reef. /avoid  common  in  on?  3  «  No 


44.  D«*W  *M.h  PRACTICE  REMPMhf.lt INC 
item  have  the  v*me  content  and 
format  at  the  te»t  item" 


i  •  Yea 

I  •  Step  »i»  it  too  unall. 
3  •  Step  tire  it  too  |ir(«. 


1  »  Yet 

2  •  Slightly  out  of  sequence 

3  v  Vary  different  teqoance 


I  .  LN.rd 
3  ■  Nor,  u*ed 


t  *  Same 

7  *  Sam*  eentent,  different  forma? 
3  i  Different  Ctv.tent 


1  rterwc\cx  war  u  gu**. 

2  *  Correct  a^twer  «n|y  ia  given. 

3  *  No  leviShfttk  b  given. 


I  *  ltrt)»  give*  tvpianati  **. 

7  *  Help  ghrt  tntuffkwnt 
etplonatvcau 
1  .  Ifctp  n  cvnlmang. 


|  «  U.nt  o.tvfU  met 
1  i  Several  (ninia  M  met 
3  *  Pew  fcfrttvL*  met 


t  *  Yel*  tew  h wd  ward*  a*4 

kn*  kr»*»«k*  ,'k 

2  *  V/e  hifd  wur«h  and  |<ng 

aet*«em.\-k 

I  ,  M*  ,f  fuivf  wwh  a*J  Lvirg 
tee.*-**  n 


*  Ye* 

-  IVj'!  *r-J  S*WW*tr*k»«» 

*  tl«l  j<*  *  iicn  M 


Mwaimainirninfli - - — — aMia 


it.v%vw  w.at.'-’.Vi. 


VVJU*A1.VAV.  A'A  VA*JW,AVVAVV_VA\.VA'A’! 


WORKSHEET  Cl.l  (continued) 
Date: 


LESSON  Title 


TEE  QUESTION 

RATING  (Circle  the  number) 

73.  Is  the  Instructor's  presentation  or 
the  narration  supported  by  visuals? 

1  *  Completely 

2  =  Partially 

3  =  Not  at  all 

7a.  Are  visuals  easily  understood? 

1  *  Yes 

2  >  Understandable  with  effort 

3  s  Very  hard  to  understand 

*73.  Are  the  OBJECTIVES  'TLOs  and 

LOs)  within  each  LESSON 
sequenced  properly?  (Prerequisites 
taught  first.) 

1  »  Yes 

3  «  No 

SO.  Do  course  administration  directions 
make  realistic  demands  of  students 
and  instructors? 

1  a  AM  demands  are  realistic. 

3  a  Some  demands  are  unrealistic. 
(Note  what  they  are) 

St.  Is  the  instructor/trainee  ratio  such 
that  all  students  can  see,  hear,  and 
receive  feedback? 

1  *  Yes 

2  «  A  few  students  cannot  see, 

hear,  and  receive  feedback. 

3  *  Many  students  cannot  see, 

hear,  end  receive  feedback. 

82.  Does  the  instructor  follow  the 
method?  in  the  Instructor  Guide? 

1  -  Yes 

2  »  Follows  to  *ome  extent 

3  *  Follows  very  little  or  not 

at  all 

S3.  Does  the  instructor  teach  all  of  the 
content  In  the  LESSON  materials? 

1  »  Yes 

2  •  Much  of  the  content 

3  ■  Very  little  of  the  content 

01  2  or  3,  note  what  was 
left  out.) 

S«*.  Did  the  instructor  limit  hit  teaching 
to  the  content  in  the  LESSON 
materials? 

1  ■  Yes 

3  3  No  (Please  note  what  other 
things  he  taught.) 

S3.  Is  there  enough  space  for  all  of  the 
trainees? 

1  a  Yes 

2  *  A  little  crowded 

3  «  Very  cramped  or  some 

students  can't  tit  in  the 
space  at  all. 

86.  Is  instruction  free  of  distractions? 

1  i  Yes 

2  «  Directions  are  annoying. 

3  •  Distractions  seriously 

interfere  with  the  instruction. 

87.  Is  the  lighting  appropriate  for  the 
training  situation? 

1  v  Yes 

2  »  Students  have  trouble  reading  or 

seeing  displays  and  equipment. 

3  »  Students  cannot  read  or  ».ee 

dispUys  and  equipme?>t. 


IS.  is  the  temperature  appropriate  for 

1  a  Yes 

training  situation? 

2  »  Temperature  makes  st«jdenti 

oncornfnrtjhlr*. 

3  a  Temperature  seriously  interferes 

with  learning. 

S3.  H  the  instructor's  attitude  positive 


»  Yet 
*  Kn 


WORKSHEET  Cl.l  (continued) 


Date: 


LESSON  Titles 


TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

90.  Are  frequent  breaks  provided? 

C5-10  minute  break  every  hour) 

1  s  Yes 

2  s  Breaks  too  short  or  infrequent 

3  =  Breaks  not  provided 

91.  b  the  speed  of  presentation 
appropriate? 

1  *  Yes 

2  a  Too  slow 

3  s  Too  fast 

92.  Was  the  allotted  training  time  too 
long  or  too  short? 

1  a  Appropriate  length 

2  «  Too  long 

3  »  Too  short 

93.  Does  the  training  devioe/equipment 
used  In  training  function  properly? 

1  •  Yes 

2  k  Minor  malfunctions,  little 

change  from  intended  task 
performance 

3  *  Major  malfunctions, 

substantial  change  from 
intended  task  performance 

99.  Is  there  anything  else  unusual  about  the  lesson  materials,  or  do  any  other 
critical  incidents  occur  during  training  that  would  interfere  with  learning? 

(Describe  each  one  below.  Rating  =  3) 

•  »  THE  analyst  only 
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WORKSHEET  C1.2:  OBSERVATION  OF  DEMONSTRATIONS 


Date: _ 

Data  Collector: _ 

Group/MOS  #:  _ _  Course: _ _ _ 

LESSON  _  LESSON  Title: - 

Applicable  TLO/LO//(s): _ 

Training  Location/Environment: _ _ 

Starting  Time: _ Instructor  ID//(s): _ 

If  any  of  the  questions  below  are  not  applicable  to  the  training  event  you  are  observing, 
write  N/A  in  the  description  column. 

Make  one  rating  for  each  question.  If  several  things  could  be  rated  for  a  question, 
take  them  all  into  account  in  making  one  rating  for  the  event. 


TEE  QUESTION 

RATING  (Orel*  the  number) 

DESCRIPTION 

2?.  Are  motivational  techniques  employed? 

1  »  Yes  (Please  describe  them.) 

3  »  No 

2S*  b  the  trainee  attitude  positive? 

1  «  Positive 

2  b  Indifferent 

J  •  Hostile  nr  frustrated 

29.  Is  mastery  of  prerequisite  skills 
verified  prior  to  new  instruction? 

1  »  Yes 

2  -  No 

}|.  Are  OBJECTIVES  presented  to  the 
student? 

1  *  Yes 

9  .  No 

U.  Does  training  Include  instruction  on 
the  use  of  required  Jab  performance 
aids? 

1-  Yes 

3  •  No 

•>$.  Is  EXAMPLE  HELP  adequate? 

1  ■  Help  provide  sufficient 

explanation. 

2  «  Help  gives  Insufficient 

e«p!a;vstion. 

9  ■  Help  h  confuting. 

•*&  Are  there  nourfi  EXAMPLES?  (See 

Job  aid  fur  criteria.) 

1  •  Yes 

9  «  No 

4?.  DO  DEMONSTRATIONS  show  huw  to 
correct/avoid  common  errors? 

1  .  Yet 

9  >  No 

•  *  TEC  A**tyst  only 


WORKSHEET  C1.2  (continued) 
Date: 


LESSON  Title 


TEE  QUESTION 

RATING  (Circle  the  number) 

*43.  Arc  step*  in  a  DEMONSTRATION 
the  appropriate  size?  (See  job  aid.) 

1  =  Yes 

2  =  Step  size  is  too  small. 

3  e  Step  size  is  too  large. 

44.  Are  task*  and  task  steps 

DEMONSTRATED  in  the  same 
sequence  as  they  are  performed  in 
the  real  world? 

1  =  Yes 

2  =  Slightly  out  of  sequence 

3  --  Very  different  .sequence 

Are  memory  aids  used?  (PRACTICE 
REMEMBERING  only) 

1  «  Used 

3  =  Not  used 

46.  Docs  each  PRACTICE  REMEMBERING 
Item  have  the  same  content  and 
format  as  the  test  item? 

1  *  Same 

2  -  Same  content,  different  format 

3  «  Different  content 

70.  b  the  technical  quality  of  written 
or  spoken  material  adequate?  (See 

Job  aid.  for  criteria.  Make  notes  on 
specific  problems.) 

f  i  Most  criteria  met 

2  a  Several  criteria  not  met 

3  »  Few  criteria  met 

71.  h  site  wording  of  written  or  spoken 
material  easy  lor  the  students  to 
understand? 

t  ■  Yes,  few  hard  words  and 

Ion*  sentence* 

2  »  Some  hard  words  and  long 

sentences 

3  a  Many  hard  word*  and  long 

sentences 

72,  b  the  instructor'!  presentation  or 
the  narration  easy  to  listen  to? 

1 

1  »  Yes 

2  »  Dull  and  monotonous 

3  ■  Hard  to  iistro  to 

73.  Is  the  instructor'!  presentation  or 
the  narration  supported  by  vUuats? 

1  *  Completely 

2  *  Partially 

3  ■  Not  at  all 

•73.  Are  the  OBJECTIVES  (TLCs  and 

LO»>  wlthm  each  LESSON 

**<*  encH  properly?  (Prerequisite* 

!*«;;•*  lint) 

!  s  Yes 

3  =  No 

SO.  Do  course  administration  directions 
make  realistic  demands  of  students 
and  instructors? 

1  •  All  demand*  are  realistic. 

3  *  Some  demand*  are  unrealistic, 
(Note  what  they  are.) 

It.  1*  the  mart*  tor/traUvv  ratio  such 
that  students  can  see,  hear,  *»d 

receive  feedback** 

1  «  Ye* 

7  «  A  few  student*  cannot  *ee, 
hear,  and  receive  feedback. 

3  *  Ma^y  student*  rannet  see, 
hear,  and  receive  feedback. 

12.  Does  the  instructor  follow  tv 
•tettwA  tn  the  instructor  Guide"* 

1  '  Ye* 

2  f  Poilows  *n  jkvt»«  extent 

3  »  Follow*  wry  latte  or  not 

at  all 

IV  !>&n  the  imtructor  tear*!  all  of  »te* 
Content  in  the  LtSbOS'  maieritli* 

1  *  Ye*  ! 

?  ■  Mirk  of  the  irntm* 

3  «  Very  fifth*  rf  :l«*  content 
(If  2  or  3,  note  wtut  wa* 
kft  out 3 

WORKSHEET  CI.2  (continued) 
Date: 


LESSON  Title: 


TEE  QUESTIOH 

RATINC  (Circle  the  number) 

84.  Did  the  instructor  limit  his  teaching 
to  the  content  in  the  LESSON 
materials? 

1  =  Yes 

3  a  No  (Please  note  what  other 
things  he  taught.) 

85.  Is  there  ertough  space  for  a!!  of  the 
trainees? 

1  a  Yes 

2  s  A  little  crowded 

3  »  Very  cramped  or  some 

students  can't  fit  in  the 
space  at  ail. 

56.  Is  instruction  free  of  distractions? 

1  ■  Yes 

2  «  Distractions  are  annoying. 

3  *  Distractions  seriously 

interfere  with  the  instruction. 

8?.  Is  the  lighting  appropriate  lor  the 
training  situation? 

1  «  Yes 

2  t  Students  have  trouble  reading  or 

seeing  displays  and  equipment. 

3  *  Students  cannot  read  or  see 

displays  and  equipment. 

88.  to  the  temperature  appropriate  for 
training  situation? 

1  »  Yes 

2  ■  Temperature  makes  students 

uncomfortable. 

3  ■  Temperature  seriously  Interferes 

with  learning. 

84.  is  the  instructor**  attitude  positive? 

W  •• 

II  • 

5? 

40.  Are  frequent  breaks  provided? 

(V10  minute  break  every  hour) 

1  a  Yes 

2  *  Breaks  too  short  or  infrequent 

3  »  breaks  not  provided 

41.  to  the  speed  of  presentation 
appropriate? 

1  n  Yes 

2  »  Too  slow 

3  *  Too  fast 

42.  Was  the  allotted  training  time  too 
long  or  too  short? 

1  ■  Appropriate  length 

2  •  Too  long 

3  •  Too  short 

45.  Does  .  ;  training  device/equipment 
used  n  training  function  properly? 

1  *  Yes 

2  *  Minor  malfunctions,  little 

change  from  intended  task 
performance 

3  w  Major  malfunctions, 

substantial  change  from 

Intended  task  performance 

44.  to  there  anythin-  el**  immi«|  about  the  Imw  materials,  or  A»  any  other 
rriiieil  LvivVnti  ertuf  during  training  that  would  Interfere  w«lK  learning? 
(Describe  each  ©ne  below.  Hating  *  5) 


•  •  TEE  analyst  only 


WORKSHEET  C1.3:  OBSERVATION  OF  ISOLATED  PRACTICE  (USING) 


Date:  —  - ..  . 

Data  Collector: _ 

Group/MOS  //: _ 1 _  Course: _ 

LESSON  -  LESSON  Title: _ 

Applicable  TLO/LO//(s): _ ! _ 

Training  Location/Environment: _ 

Starting  Time: _ Instructor  ID//(s):  _ 

Ending  Time: _ _ 

Individual  □  Crew  □ 

If  any  of  the  questions  belt  are  not  applicable  to  the  training  event  you  are  observing, 
write  N/A  in  the  description  column. 


Make  one  rating  for  each  question.  If  several  things  could  be  rated  for  a  question, 
take  them  all  into  account  in  making  one  rating  for  the  event. 


_ TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

27.  Are  motivational  technique*  employed? 

1  •  Yes  (Please  describe  them.) 

3  >  No 

2S.  b  the  trainee  attitude  positive? 

1  s  Positive 

2  «  Indifferent 

3  *  Hostile  or  frustrated 

* 

J3.  b  mastery  of  prerequisite  skills 
verified  prior  to  new  instruction? 

1  •  Yes 

2  >  No 

3!.  Are  OBJECTIVES  presented  to  the 
student? 

1  •  Yes 

3  «  No 

36.  Does  training  include  instruction  on 
the  use  pi  required  Job  performance 
aids? 

1  ■  Yes 

3  .  No 

*7.  Any  PRACTICE  USING  items 
•equmced  from  easy  to  hard? 

1  •  Yes 

3  -•  No 

•61.  Do  PRACTICE  USING  items  provide 
opportunities  for  common  errors 
to  be  made? 

1  •  Yes 

3  *  No 

•  •  TEE  Analyst  only 


WORKSHEET  C1.3  (continued) 


Date: 


LESSON  Title: 


TEE  QUESTION 


30.  Does  PRACTICE  USING  integrate 
tasks  as  they  are  integrated  in 
the  "real  world?" 


31.  Are  job  performance  aids 
CPAs)  usable? 


32.  Do  ail  students  use  the  Job 
performance  aid  (3 PA)? 


33.  Does  the  TASK  LEVEL  of  the 
item  match  that  of  the  test 
item(s)? 


33.  Does  the  FORMAT  oi  the 
PRACTICE  item  match  that  of 
the  test  itemfc)? 


5*.  Do  the  CONDITIONS  of  each 
FINAL  PRACTICE  Item  match 
those  of  the  test  item(s)? 


37.  Do  the  STANDARDS  of  each 
FINAL  PRACTICE  item  match 
those  of  the  test  ItemCs)? 


32.  Is  FINAL  PRACTICE  free  of 
external  cues  or  hMp? 


41.  Do  ait  students  PRACTICE? 


€2.  Do  ali  students  meet  the 
reared  STANDARDS  in 
FINAL  PRACTICE? 


43.  Is  FEEDBACK  provided  (or 
PRACTICE? 


RATING  (Circle  the  number) 


1  =  Yes 

2  =  Partially 

3  ■  No,  tasks  are  practiced  separately. 


1  s  Easy  to  use 

2  s  Hard  to  use 

3  *  Unusable 


1  «  Yes 

2  =  Up  to  20%  do  not  use  3PA. 

3  *  More  than  20%  do  not  use  3PA. 


1  -  Yes 

(2  and  3  —  See  table  in  job  aid.) 


I  r  Yes 

(2  and  3  —  See  table  in  job  aid.) 


1  -  Yes 

2  m  Slightly  different 

3  -  Very  different 


1  V  Yes 

2  -  Slightly  different 

3  ■  Very  different 


1  -  Yes 

2  *  Hints  given 

3  «  Answers  are  given  away. 


1  .  Yes 

2  «  Up  to  20%  of  students  do  not 

PRACTICE. 

3  *  More  than  20%  of  students  do 

not  PRACTICE. 


1  •  Yes 

2  ■  Up  to  20%  of  studrtus  do  not. 

3  More  than  20%  of  students  do  not . 


1  k  FEEDBACK  HELP  is  given. 

2  •  Correct  aruv-er  only  is  given. 

3  -  No  FEEDBACK  is  give*. 


WORKSHEET  C1.3  (continued) 


Date: 


LESSON  Title: 


TEE  QUESTION1 

RATING  (Circle  the  number) 

*6ft.  IS  FEEDBACK  HELP  Adequate? 

1  »  Help  gives  enough  explanation. 

2  =  Help  gives  insufficient 

explanation. 

3  »  Help  Is  confusing. 

70.  Is  the  technical  quality  of  written 
or  spoken  material  adequate?  (See 

Job  aid  for  criteria.  Make  notes  on 
specific  problems.) 

1  *  Most  criteria  met 

2  a  Several  criteria  not  met 

3  *  Few  criteria  met 

71.  Is  the  wording  of  written  or  spoken 
material  easy  for  the  students  to 
understand? 

1  *  Yes,  few  hard  words  and 

long  sentences 

2  *  Some  hard  words  and  long 

sentences 

3  =  Many  hard  words  and  long 

sentences 

7ft.  Are  visuals  easily  understood? 

1  «  Yes 

2  s  Understandable  with  effort 

3  3  Very  hard  to  understand 

*75.  Are  the  OBJECTIVES  (TLOs  and 

LOs)  within  each  LESSON 
sequenced  properly?  (Prerequisites 
taught  first.) 

1  .  Yes 

3  ■  No 

80.  Do  course  administration  directions 
make  realistic  demands  of  students 
and  instructors? 

1  •  All  demands  are  realistic 

3  ■  Some  demands  -re  unrealistic. 
(Note  what  they  are.) 

11.  b  the  imtructor/tralnce  ratio  such 
tlwt  all  students  can  see,  hear,  and 
receive  feedback? 

1  ■  Yes 

2  ■  A  few  students  cannot  see, 

hear,  and  receive  feedback. 

3  a  Many  students  cannot  see, 

hear,  and  receive  feedback. 

87.  Does  the  instructor  follow  the 
methods  In  the  Instructor  Guide? 

1  •  Yes 

2  a  Pollows  to  some  extent 

3  w  Fellows  very  little  or  not 

at  all 

83.  Does  the  instructor  teach  all  of  the 
content  In  the  LESSON  materials? 

1  .  Yes 

2  a  Much  of  the  content 

3  a  Very  little  of  the  content 

(If  2  or  3,  note  that  was 
left  out) 

3ft.  Did  the  instructor  limit  his  teaching 
to  the  content  In  the  LESSON 
material*? 

1  «  Yes 

3  a  No  (Please  note  what  other 
things  he  taujhtj 

83.  Is  there  enough  space  for  alt  of  the 
trainees? 

1  a  YCS 

2  a  A  little  crowded 

3  a  Very  cram***!  or  some 

students  can’t  fit  in  tlve 
space  at  all. 

8ft.  h  instruction  free  of  distractions? 

1  S  Ye* 

2  a  Distractions  are  annoying. 

3  a  Distractions  seriously 

inter fe<e  with  the  Instruction. 

8?.  Is  the  lighting  appropriate  for  the 
training  situation? 

1  a  YCS 

2  *  Studr.n  *  rave  trouble  reachng  or 

seeing  displays  and  Cvftipstie.il. 

3  *  StuJcnu  cannot  read  «■  s«e 

display*  end  equipment. 

DESCRIPTION 


*  *  TEE  only 


WORKSHEET  C1.3  (continued) 


Date: 


LESSON  Title: 


TEC  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

SS.  b  the  temperature  appropriate  for 
training  situation? 

1  *  Yes 

2  »  Temperature  makes  students 

uncomfortable. 

3  *  Temperature  seriously  interferes 

with  learning. 

W*  U  the  instructor's  attitude  positive? 

1  Yes 

3  »  No 

90.  Are  frequent  breaks  provided? 

(5-10  minute  break  every  hour) 

1  *  Yes 

2  ■  Breaks  too  short  or  Infrequent 

3  »  Breaks  not  provided 

92.  Was  the  allotted  training  time  too 
long  or  too  short? 

1  «  Appropriate  length 

2  ■  Too  long 

•  3  «  Too  short 

53,  Does  the  training  devicc/equipment 
used  in  training  function  properly? 

1  ■  Yes 

2  »  Minor  malfunctions,  little 

change  from  intended  task 
performance 

3  «  Major  malfunctions, 

substantial  change  from 
intended  task  peflwuwnCc 

9*.  b  there  anything  else  unusual  about  the  lesson  materials,  or  do  any  other 
critical  Incidents  occur  during  training  that  would  Interfere  with  (earning? 

(Describe  each  one  below.  Rating  •  J) 

•  «  TEC  anatyt:  only 
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WORKSHEET  C1.4:  OBSERVATION  OF  A  PERFORMANCE  TEST 


Date:  ..  • _ 

Data  Collector:  _ 

Group/MOS  //:  Course: _ 

Type  of  Test:  LESSON  □  (LESSON  //): _ Final  Q 

Individual  □  Crew  □  Other _ 

Applicable  TLO/LO  //s: _ _ _ 

Test  Location/Environments _ ! _ 

Starting  Time: _ Test  Administrator  (Instructor)  ID  #(s): _ 

Ending  Time: _ _ Test  Scorer  (Instructor)  ID  //(s): _ 

If  any  of  the  questions  below  are  not  applicabale,  write  N/A  in  the  Description  column. 

Make  one  rating  for  each  question  below.  If  several  things  could  be  rated  for  a 
question,  take  them  all  into  account  in  making  one  rating  for  the  test.  If  applicable, 
however,  note  which  test  items  apply  in  the  Description  column.  In  a  performance 
test,  use  a  new  worksheet  each  time  the  instructor  directs  the  students  to  perform  a 
new  activity. 


TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

ft.  Do  th«  CONDITIONS  of  the  test 
hem  match  tho  CONDITIONS  of 

Its  OBJECTIVE? 

1  a  Exact  match 

2  •  Minor  mismatch 

3  a  Severe  mismatch 

7.  Do  the  STANDARDS  of  the  test 

Item  match  the  STANDARDS  of 

IU  OBJECTIVE? 

1  •  Ewe!  match 

2  a  Minor  mismatch 

3  a  Severe  mismatch 

• 

•10.  Doe*  the  test  Item  provide 

opportunities  for  common  error* 
to  be  made? 

1  •  Yes 

3  «  No 

11.  Is  the  language  of  the  test  Item 
easy  for  students  to  understand? 

1  »  Easy 

2  *  Semenrhat  difficult 

3  .  Very  dillkvH 

ift*  It  the  test  Htm  tricky  or 
ml»S*a<&t£? 

t  *  Not  misleading 

2  »  Somewhat  misleading 

3  *  Very 

IS.  ftsrft  formant*  steps  art  scored, 
does  the  instructor  w*  • 

dseJkjHt? 

t  e  nil*  in  fonplrlrlf 

2  •  u*e*  M  a  rrl«i“we  or  fiiit 

m  partially 

3  r  Doc.  rot  uw 

71.  Are  te«  r^lnnirtiian 
ftfrctiJRi  compi-le' 

I  *  Direct mv  c&hpU-'.v 
?  .  Dtrtx't»*»  h»»t 

W  iirr.plr;*  t * 

3  *  Dtr*<t»ar*  art  nr*t  provided. 

•  »  TEC  «nt$  H-J8 


r.  «v  •  _  c*  o-u  rw  mu  w~-  ■*_  m  u  •»  -  Wm  mu1 


TEE  Aru?/i!  wljr 
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WORKSHEET  C  1.5:  OBSERVATION  OF  A  WRITTEN  TEST 


Date: _ 

Data  Collector: _ 

Group/MOS  if: _ Course: _ 

Type  of  Test:  LESSON  □  (LESSON  if): _  Final  □ 

Individual  □  Crew  □  Other  □ _ 

Applicable  TLO/LO  ih • _ 

Test  Location/Environment: _ _ 

Starting  Time; _ Test  Administrator  (Instructor)  ID  #($): _ 

Ending  Time:  _ _ Test  Scorer  (Instructor)  ID  if(s):  ....  ..  _ 

If  any  of  the  questions  below  arc  not  applicabale,  write  N/A  in  the  Description  column. 

Make  one  rating  for  each  question  below.  If  several  things  could  be  rated  for  a 
question,  take  them  all  into  account  in  making  one  rating  for  the  test.  If  applicable, 
however,  note  which  test  items  apply  in  the  Description  column. 


TEE  QUESTION 

RATING  (Circle  the  number) 

21.  Art  leu  »droM»tr»tU*i 
direction*  complete? 

t  •  Direction*  jtr*  complete. 

2  *  Direction*  provided,  hut 
iiKOmptete  or  unclear 

1  •  Directions  err  net  provided. 

22.  bo  intmtetor*  fot|o%>  the  direction* 
when  inj  the  tett? 

1  *  Ye« 

2  *  Some  v^fUtlOn*  from  drettiortt 

3  »  Significant  variations  from 

direct  ton* 

21.  Are  »det*Mt»  t*»t  in*truett©n» 
provided  to  the  student? 

1  •  Yet 

?  «  Insmtr'taiu  provided,  but  unclear 

1  .  No  ln>truetion*  provided 

24.  H  the  test  free  of  external  cce* 
m  help? 

1  *  Yet 

2  #  Hinti  given 

3  *  Answer*  ore  g»wn 

W.  b  the  test  free  e(  dtsuKtWftf* 

1  «  Yev 

2  .  Ortvactkim  *»c  annoying. 

)  *  mutual**!  MnOuvly 

interfere  wjlh  the  |**f. 

27.  h  th*  lighting-  appropriate  fee  the 
MtwPtiort? 

1  *  Ya 

2  •  Studrittt  have  trouble  re*c’rt*g. 

)  .  StuAnit*  tdtwot  re  id. 

tft.  h  the  ten.ptf*tyfe  for 

!*»*>•*$  S*tu»t»os? 

1  ♦  Yr» 

UWCJVntft^uHle. 

wtlh  tK<*  tn»t. 

•  •  TU  only 


H-  20 


ILk f j» E* ?-* fj  fj. •  * J  V *-•  ^ 


WORKSHEET  C1.5  (continued) 

Dates  LESSON  Titles 


TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

19.  h  the  instructor's  attitude  positive? 

1  *  Yes 

5  »  No 

- 

99.  Are  .'request  breaks  provided? 

(5-19  minute  break  every  hour! 

1  •  Yes 

2  a  Breaks  too  short  or  infrequent 

5  »  Breaks  not  provided 

92.  Wu  the  allotted  testing  tint*  too 
lon(  or  too  short? 

1  «  Appropriate  length 

2  >  Too  long 

5  ■  Tea  short 

H  Is  there  anything  else  unusml  about  the  lesson  mater  iafs,  or  do  »ny  other 
crltlcAj  incidents  occur  during  testing  thet  would  interfere  with  learning? 

(Describe  each  one  below.  Rating  »  )) 

•  »  TEE  analyst  onl/ 


WORKSHEET  C1.6:  OBSERVATION  OF  AN  ORAL  TEST 


Date:  _ _ 

Data  Collector: _ 

Group/MOS  #: _ Course: _ _ 

Type  of  Test:  LESSON  □  (LESSON  fi)i _  Final  □ 

Individual  □  Crew  Q  Other  IT 

Applicable  TLO/LO  tfs:  _ 

Test  Location/Environment:  _ _ _ _ 

Starting  Time:  T««t  Administrator  (Instructor)  ID  0(s): _ „ 

Ending  Time:  _ Test  Scorer  (Instructor)  ID  #(s): _ _ 

If  any  of  the  questions  below  are  not  applicabale,  write  N/A  in  the  Description  column. 

Make  one  rating  for  each  question  below.  If  several  things  could  be  rated  for  a 
question,  take  them  all  into  account  in  making  one  rating  for  the  test.  If  applicable, 
however,  note  which  test  items  apply  In  the  Description  column.  In  a  performance 
test,  use  a  new  worksheet  each  time  the  instructor  directs  the  students  to  perform  a 
new  activity. 


THE  QUESTION 

RATING  (Cert*  the  iWw) 

nesoutmoN 

*.  Do  the  CONDITIONS  of  the  t<m 
Hem  much  the  CONDITIONS  H 

Ha  OBJECTIVE? 

1  •  Eeart  hteteh 

I  *  Mine*  mwtutch 

S  »  Severe 

7.  Do  *h*  STAND  \RDS  of  the  mi 

Hem  the  STANDARDS  of 

h»  owccnvi? 

1  *  CtKt  mvch 

2  4  ilimr  mismatch 

J  »  Sever*  mtunjtch 

It*  ts  the  Ur$*t£»  ttf  the  tfiX  hem 
ooty  for  ttoOroff  to  \r^ca\i  ~ 

1  «  R»y 

I  *  Sambl  dik^l 

S  *  Very  difficult 

1%.  U  the  wt  to  th-  mi  item 

*  t  onweerUei  frevimj* 

Hemb!  eurwetiy? 

I  *  Amwtf  M  stile 

ttvmt 

J  *  item.  he 

IS.  Ar#  m4  i»»N  In 

th»  t**t  he*  e**y  to 

1  *  Ret*  to 

2  *  rwtv*u< 

)  *  Very 

It,  item  Uicfcy  er 

|  *  Nm 

2  *  Srsmr»-H*t 
)  *  Vet 

II,  Am  »n! 

*  HtfeCttwn  eemyileivO 

1  t  A»e  . 

2  4  (IV hut 

1  *  M 

*  •  TEC  er'Jy 


jjtv> 


WORKSHEET  C1.6  (continued) 
Date: 


LESSON  Title: 


TEE  QUESTION 

RATING  (Circle  the  number) 

22.  Do  instructors  follow  the  directions 
when  administering  the  test? 

1  *  Yes 

*2  x  Some  variations  from  directions 

3  »  Significant  variations  from 
directions 

23.  Are  adequate  test  instructions 
provided  to  the  student? 

1  =  Yes 

2  =  Instructions  provided,  but  unclear 

3  a  No  instructions  provided 

26.  Is  the  test  free  of  external  cues 
or  help? 

1  =  Yes 

2  a  Hints  given 

3  a  Answers  are  given  away. 

66.  Is  the  test  free  of  distractions? 

1  a  Yes 

2  «  Distractions  are  annoying. 

3  a  Distractions  seriously 

Interfere  with  the  test. 

*7.  is  the  lighting  appropriate  for  the 
testing  situation? 

1  a  Yes 

2  «  Students  have  trouble  reading  or 

seeing  displays  and  equipment. 

3  «  Students  cannot  read  or  see 

displays  and  equipment. 

86.  Is  the  temperature  appropriate  for 
testing  situation? 

1  «  Yes 

2  v  Temperature  makes  students 

uncomfortable . 

3  *  Temperature  seriously  Interferes 

with  the  test. 

5?,  Is  the  Instructor’s  attitude  positive? 

I  >  Yes 

3  ■  No 

90.  Are  frequent  breaks  provided? 

(3-10  minute  break  every  hour) 

1  «  Yes 

2  r  Breaks  too  short  or  Infrequent 

3  a  Breaks  not  provided 

92.  Was  the  allotted  testing  time  too 
long  or  too  short? 

1  ■  Appropriate  length 

2  *  Too  long 

3  »  Too  short 

91.  Does  the  training  device/equipment 
used  In  testing  function  properly? 

1  •  Yes 

2  *  Minor  malfunctions,  little 

change  from  Intended  task 
performance 
)  a  Major  malfunctions, 
substantial  change  from 
intended  task  performance 

9 *•*  Is  there  anything  else  uousu  I  about  the  lesson  materiiilv,  or  do  any  other 
critics!  incidents  occur  Airing  testing  that  would  Interfere  with  learning? 
(Describe  each  one  below.  Rating  ■  3) 


DESCRIPTION 


*-'*v*\i*«  v*\.v  wjyw>  k>~v»vvy->  w.> 


WORKSHEET  0.7:  OBSERVATION  OF  INTEGRATED  PRACTICE 


Date:  _ _ _ 

Data  Collector:  _ 

Group/MOS  #: _ _ _ Course: _ _ _ _ 

LESSON  #: _ LESSON  Title: _ 

Applicable  TLO/LO  //(s): _ _ _ 

Training  Location/Environment: _ 

Starting  Time: _ Instructor  ID  //(s): _ 

Ending  Time: _ * _ _ 

Individual  □  Crew  □ 

If  any  of  the  questions  below  are  not  applicable  to  the  training  event  you  are  observing, 
write  N/A  in  the  Description  column. 

Make  one  rating  for  each  question.  If  several  things  could  be  rated  for  a  question, 
take  them  all  into  account  in  makL  ^  one  rating  for  the  event. 


TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

27.  Are  .motivational  techniques  employed? 

1  ■  Yes  (Please  describe  them.) 

3  •  No 

21  Is  the  trainee  attitude  positive? 

1  •  Positive 

2  ■  Indifferent 

3  ■  Hostile  or  frustrated 

25.  Are  course  ENTRY  SKILLS 
reviewed? 

1  «  Review  with  practice 

2  ■  Review  with  no  practice 

3  a  No  review 

10.  Is  mastery  ol  prerequisite  skills 
verified  prior  to  new  Instruction? 

1  •  Yes 

2  *  No 

31.  Are  OBJECTIVES  presented  to  the 
student? 

1  .  Ye* 

3  «  No 

•n.  Do  PRACTICE  USING  Item*  provide 
opportunities  (or  common  errors 
to  be  made? 

1  «  Yes 

3  <  No 

SO.  Doe*  PRACTICE  USING  Integrate 
tasks  at  they  are  Integrated  m 
the  "real  world*" 

1  ^  Yes 

2  *  Partially 

3  a  No,  tasks  are  practiced  separately. 

•  *  TEE  Analyst  only 


WORKSHEET  C1.7  (continued) 
Date: 


LESSON  Title 


TEE  QUESTION 

•  RATING  (Circle  the  number) 

SI.  Are  job  performance  aids 

OPAs)  usable? 

1  a  Easy  to  use 

2  =  Hard  to  use 

3  j  Unusable 

52.  Do  til  students  use  the  job 
performance  aid  (JPA)? 

1  »  Yes 

2  s  Up  to  20%  do  not  use  DPA 

3  *  More  than  20%  do  not  use  3PA 

S3.  Does  the  TASK  LEVEL  of  the 
PRACTICE  item  match  that 
of  the  test  item(s)? 

1  «  Yes 

(2  and  3  ~  See  table  in  job  aid.) 

55.  Does  the  FORMAT  ot  the 

PRACTICE  item  match  that  of 
the  test  item(s)? 

1  »  Yes 

(2  and  3  —  See  table  in  job  aid.) 

56.  Do  the  CONDITIONS  of  each 

FINAL  PRACTICE  Item  match 
those  of  the  test  item(»)? 

1  a  Yes 

2  •  Slightly  different 

3  a  Very  different 

57.  Do  the  STANDARDS  of  each 

FINAL  PRACTICE  item  match 
those  of  the  test  item(s)? 

1  r  Yes 

2  •  Slightly  different 

3  ■  Very  different 

>1.  Is  FINAL  PRACTICE  free  of 
external  cues  or  help? 

1  a  Yes 

2  ■  Hints  given 

3  a  Answers  are  given  away. 

<1.  Do  ell  students  PRACTICE? 

1  *  Yes 

2  a  Up  to  70%  of  students  do  not 

PRACTICE. 

3  a  More  than  20%  of  students  do 

not  PRACTICE. 

67.  Do  all  students  meet  the 
required  STANDARDS  In 

FINAL  PRACTICE? 

1  a  Yes 

2  a  Up  to  20%  of  students  do  not. 

3  •-  More  than  20%  of  students  cto  not. 

«.  Is  FEEDBACK  provided  for 
PRACTICE? 

1  a  FEEDBACK  HELP  tJ  given. 

2  *  Co;r*-ct  answer  only  is  given. 

3  •  No  FEEDBACK  Is  given. 

Hk.  IS  FEEDBACK  HELP  adequate? 

1  a  Help  gives  enough  explanation. 

2  a  Help  gives  insuflincnt 

explanation. 

3  a  Help  is  confuting. 

65.  Is  TEAM  FRACTICE  provided? 

1  S  Yes 

3  .  No 

»  TEE  Analyst  only 


WORKSHEET  C1.7  (continued) 

Dates  LESSON  Titles 


TEE  QUESTION 

RATING  (Circle  the  number) 

66.  Are  TEAM  PRACTICE 

CONDITIONS  la  the  same  as  (or 
as  close  as  possible  to)  those  of 
the  real  task? 

1  r  Yes 

2  *  Slightly  different 

3  =  Vtry  different 

6?.  b  TEAM  PRACTICE  FEEDBACK 
provided? 

1  r  FEEDBACK  HELP  is  given. 

2  =  Success/Failure  feedback  only 

is  given . 

3  «  No  FEEDBACK  U  given. 

*68.  b  FEEDBACK  HELP  lor  TEAM 
PRACTICE  adequate? 

1  *  Help  gives  enough  explanation. 

2  *  Help  gives  Insufficient  explanation. 

3  *  Help  b  confusing. 

70.  b  the  technical  quality  of  written 
or  spoken  material  adequate?  (See 
job  aid  for  criteria.  Make  notes  on 
specific  problems. I 

1  «  Most  criteria  met 

2  a  Several  criteria  not  met 

3  a  Few  criteria  m^t 

71.  b  the  wording  of  written  or  spoken 
material  easy  for  the  students  to 
understand? 

1  «  Yes,  few  hard  words  and 

long  sentences 

2  «  Some  h*rd  words  and  long 

sentences 

3  »  Many  hard  words  and  long 

sentences 

72.  U  the  instructor's  presentation  or 
the  narration  easy  to  listen  to? 

1  «  Yes 

?  a  Dull  and  monotonous 

3  «  Hard  to  listen  to 

SO.  Do  course  administration  directions 
make  realistic  demands  of  students 
and  instructors? 

1  a  All  Ormands  are  realistic. 

3  ■  Some  demands  are  uareallstic. 

(Note  what  tliey  are) 

tt.  h  the  instructor/tralnre  ratio  such 
that  aii  students  can  tee,  hear,  and 
receive  feedback? 

1  vet 

2  a  a  few  students  cannot  see, 

hear,  and  receive  feedback. 

3  a  Many  students  tannut  see, 

hear,  and  receive  feedback. 

12.  Does  the  instructor  foiiow  the 
method*  in  the  Instructor  Guide? 

1  •  Yes 

2  a  Follows  to  tome  extent 

3  *  Follows  very  Idtle  or  not 

at  all 

S3.  Does  tint  instructor  teach  all  of  the 
content  in  the  LESSON  mat**  U  Is? 

1  *  Ye*. 

2  a  Much  ©!  '.be  content 

3  a  Very  .((tie  ol  the  content 

Of  2  or  3,  note  what  was 

loft  OUtJ 

14.  Did  the  instructor  limit  hit  teaching 
to  the  content  In  the  LESSON 
materia!*? 

1  a  Ye* 

3  a  No  (Ple«e  nut'1  what  other 
things  he  taught) 

85.  b  there  enough  •pace  lor  all  of  the 
train***? 

1  W  Yes 

2  *■  A  little  crowded 

3  «  Very  cramped  or  some 

students  run's  fit  in  the 
space  at  ait. 

•  ■  TEE  Analyst  only 


WORKSHEET  CI.7  (continued) 
Date: 


LESSON  Title: 


TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

S6.  Is  instruction  free  of  discretions? 

1  =  Yes 

2  s  Distractions  are  annoying. 

3  s  Distractions  seriously 

interfere  with  the  instruction. 

17.  Is  the  lighting  appropriate  for  the 
training  situation? 

1  =  Yes 

2  =  Students  have  trouble  reading  or 

seeing  displays  and  equipment. 

3  «  Students  cannot  read  or  see 

displays  and  equipment. 

U.  Is  the  temperature  appropriate  for 
training  situation? 

1  *  Yes 

2  =  Temperature  makes  students 

uncomfortable. 

3  a  Temperature  seriously  interferes 

with  learning. 

19.  Is  the  instructor's  attitude  positive? 

t  *  Yes 

3  »  No 

90.  Are  frequent  breaks  provided? 

(9-10  minute  break  every  hour) 

1  «  Yes 

2  «  Breaks  too  short  or  Infrequent 

3  »  Breaks  not  provided 

92.  Was  the  allotted  training  time  too 
long  or  too  short? 

1  *  Appropriate  length 

2  «  Too  long 

3  •  Too  short 

93.  Does  the  training  device/equipment 
used  In  training  function  properly? 

1  ■  Yes 

2  «  Minor  malfunctions,  little 

Change  from  intended  task 
performance 

3  »  Major  malfunctioni, 

substantial  change  from 

Intended  task  performance 

9*.  Is  there  anything  else  unusual  about  the  lesson  materials,  or  do  any  other 
critical  incidents  occur  during  training  that  would  interfere  with  learning? 

(Describe  each  one  below.  Rating  •  3) 

•  •  TEE  Analyst  only 
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WORKSHEET  Cl.Ss  OBSERVATION  OF  INDIVIDUAL  STUDY 


Date: _ 

Data  Collector: _ 

Group/MOS  _  Course:  _ 

LESSON  //:  _  LESSON  Title:  _ 

Applicable  TLO/LO  //(s): _ 

Training  Location/Environment: _ 

Starting  Time: _ Instructor  ID  //(s): _ 

Ending  time: _ 

If  any  of  the  questions  below  are  not  applicable  to  the  training  event  you  are  observing, 
write  N/A  in  the  Description  column. 

Make  one  rating  for  each  question.  If  several  things  could  be  rated  for  a  question, 
take  them  all  into  account  in  making  one  rating  for  the  event. 


TEE  QUESTION 

RATING  (Circle  the  number) 

DESCRIPTION 

2S.  h  the  trainee  attitude  positive? 

1  •  Positive 

2  ■  Indifferent 

3  e  Hostile  or  frustrated 

SO.  Do  course  administration  directions, 
make  realistic  demands  of  students 
and  Instructors? 

!  ■  All  demands  are  realistic. 

3  b  Some  demands  are  unrealistic. 
(Note  what  they  are.) 

SS.  h  there  enough  s^ace  for  all  of  the 
trainees? 

1  *  Yes 

2  =  A  little  crowded 

3  »  Very  cramped  or  some 

students  can't  fit  In  the 
space  at  all. 

S6.  Is  instruction  free  of  distractions? 

J  ■  Yes 

2  «  Distractions  are  annoying. 

3  ■»  Distractions  seriously 

Interfere  with  the  instruction. 

17.  Is  the  lighting  appropriate  ior  the 
training  situation? 

1  .  Yes 

2  ■  Students  have  trouble  reading  or 

seeing  dispf«ys  and  equipment. 

3  -  Students  cannot  re.id  or  see 

displays  and  etpilpment. 

JS.  Is  the  temperature  appropriate  for 
training  situation? 

!  .  Yrs 

2  »  Tempera-tire  makri  students 

uncomfortable 

3  ■  Temperature  seriously  interferes 

with  learning 

W,  Are  frequent  breaks  provided? 

{5-10  minute  break  every  hour) 

1  .  Yes 

2  «■  Break*  too  short  or  Infrequ-mt 

3  a  Dresks  not  provided 

•  ■  TEE  analyst  only  H-23 
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WORKSHEET  C1.8  (continued) 

Date:  LESSON  Title: 


TEE  QUESTION 

RATING  (Circle  tlve  number) 

DESCRIPTION 

92.  Was  the  allotted  training  time  too 
long  or  too  short? 

1  =  Appropriate  length 

2  =  Too  long 

3  s  Too  short 

• 

99.  Does  the  training  device/equipment 
used  in  training  function  properly? 

1  =  Yes 

2  s  Minor  malfunctions,  little 

change  from  intended  task 
performance 

3  *  Major  malfunctions, 

substantial  change  from 
intended  task  performance 

9ft.  Is  there  anything  else  unusual  about  the  lesson  materials,  or  do  any  other 
critical  incidents  occur  during  training  that  would  Interfere  with  learning? 

(Describe  each  one  below.  Rating  *  3) 

*  »  TEE  Analyst  only 


H-29 


Date:  _ 

Evaluator: _ 

WORKSHEET  C1.9:  RATING  OF  PROCESS  EVALUATION  QUESTIONS 
COVERING  A  SERIES  OF  TRAINING  EVENTS 

Group/MOS  in _ 

Lesson  //: _ 

Applicable  TLO/LO  //(s): _ 

Training  Location/Environments: 

Starting  Times: _ 


. . Course: 

Lesson  Title: _ 


-  Dates: - 

Data  Collectors:.. 
Instructor  ID  //(s): 


Answer  each  of  the  following  TEE  questions  by  examining  the  process  evaluation  data 
collection  worksheets  that  apply  to  the  series  of  presentation  components  for  each  task 
in  the  lesson. 


*32.  Are  the  basic  PRESENTATION 
COMPONENTS  present?  (See 
guidance  and  tables  in  Appendix  A 
for  rating.) 


*75.  Are  the  OBJECTIVES  (TLOs  I  =  Yes 
and  LOs)  within  each  3  =  No 

LESSON  sequenced 
properly?  (Prerequisites 
taught  first.) 


*76.  Are  the  LESSONS  sequenced  1  &  Yes 

properly  within  the  course?  3  r.  No  Rating 

(Have  all  of  the  prerequi¬ 
sites  for  tasks  taught  in 
this  LESSON  been  taught  in 
previous  LESSONS?) 


*  =  TEE  Analyst  Only 
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WORKSHEET  Cl.lOs  TRAINEE  REACTION  QUESTIONNAIRE 


Please  give  us  your  ideas  about  the  training  you  have  just  finished.  Be  honest.  Your 
answers  will  not  be  shown  to  anyone  in  your  command.  Please  do  not  put  your  name 
on  this  questionnaire. 


Check  how  you  feel  about  each  statement  in  the  boxes  on  the  right.  Answer  the 
question  below  each  statement. 


1.  I  enjoyed  the  training. 
Why  or  why  not? 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

□ 

□ 

□ 

□ 

□ 

Strongly  Strongly 

Disagree  Disagree  Neutral  Agree  Agree 

2.  I  was  pleased  with  rny  □  □  □  □  □ 

performance  in  the 
training. 

Why  or  why  not? 


HOI 


continued  on  next  page. 


WORKSHEET  Cl.iO  (continued) 


Page  2  of  Cl.  10 


3.  There  was  something 
about  the  training  that 
made  me  not  want  to 
learn. 


Strongly  Strongly 

Disagree  Disagree  Neutral  Agree  Agree 


List  everything  that  made  you  not  want  to  learn: 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

There  was  something  In 
the  training  that  made 
it  hard  to  learn. 

□ 

□ 

□ 

□ 

□ 

List  everything  that  made  it  hard  to  learn: 


3.  Please  make  any  other  comments  you  would  like  to  make  about  the  course. 


Date: _ 

Data  Collectors _ 

WORKSHEET  C1.II*  TRAINEE  REACTION  INTERVIEW  FORM 

Group/Mos  //: _ Course: _ 

Interview  Location/Environment: _ 

Interview  Starting  Time:  _ Instructor  ID  #(s): _ 


Ask  each  question  orally  along  with  the  five  possible  answers,  check  the  appropriate 
box,  and  record  the  main  ideas  of  the  trainee's  other  responses  in  the  space  below 
each  question. 

Be  sure  he  understands  which  training  event  you  are  asking  about. 


Strongly  Didn't  Strongly 

Disagree  Like  Undecided  Liked  Agree 


1.  How  much  did  you  like  □  □  □  □  D 

or  not  like  the  training? 

Why  or  why  not? 


Very  Very 

Unhappy  Unhappy  Undecided  Happy  Happy 

2.  How  happy  or  unhappy  □  D  □  □  D 

were  you  with  how  you 
did  in  the  training? 

Why  or  why  not? 


H-33 
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WORKSHEET  Cl.il  (Continued) 


Date  -  Lesson  Title 


Definitely 

Definitely 

Yes 

Yes 

1  Indent  led 

No 

Not 

3.  Was  there  anything  in  Q 

□ 

□ 

□ 

□ 

the  training  that 
turned  you  off  (made 
you  not  want  to  learn)? 

List  everything  that  turned  you  off: 

Definitely 

Yes 

Yes 

Under;  i«  l«'d  No 

Definitely 

Not 

4.  Was  there  anything 
about  the  training  that 
made  it  hard  to  learn? 

□ 

□ 

□ 

□ 

□ 

List  everything  that  made  it  hard  to  learn: 

5.  Please  make  any  other  comments  you  would  like  to  in.iUo  about  the  course. 


HO* 
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WORKSHEET  C1.12:  INSTRUCTOR  REACTION  QUESTIONNAIRE 


Please  give  us  your  ideas  about  the  training  event  you  have  just  conducted.  Be  honest. 
Your  answers  will  not  be  shown  to  anyone  in  your  command. 

Check  the  appropriate  box  far  each  statement  below  and  supply  the  other  information 
requested. 


Strongly 

Disagree 

l.  The  information  in  the  FI 

training  materials 
was  generally  clear. 


Disagree 

Neutral 

Agree 

Strongly 

Agree 

□ 

□ 

n 

□ 

List  tasks  for  which  material  was  unclear: 


Strongly  Strongly 

Disagree  Disagree  Neutral  Agree  Agree 

2.  Enough  information  □  □  □  □  □ 

was  provided  to 
conduct  the  course. 

List  tasks  for  which  you  needed  more  information; 


Strongly  Strongly 

Disagree  Disagree  Neutral  Agree  Agree 

3.  The  information  in  the  □  □  □  □  □ 

materia!  was  tech¬ 
nically  accurate. 

List  tasks  for  which  materials  were  inaccurate: 


H-.V* 


WORKSHEET  C1.12  (continued) 
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4.  I  had  access  to  all 
student  and  instructor 
materials  available 
for  the  course. 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

□ 

□ 

□ 

□ 

□ 

List  tasks  for  which  you  could  not  obtain  the  materials: 


5.  The  students  had  no 
trouble  learning  any 
of  the  tasks  in  the 
course. 


Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

□ 

□ 

□ 

□ 

□ 

List  the  tasks  the  students 
had  trouble  learning: 


Give  ;he  reasons  you  think  they 
had  trouble  with  each  one: 


H-3* 


WORKSHEET  CL  12  (continued) 
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6*  Please  make  any  other  comments  you  would  like  to  make  about  the  course: 


Date; _ 

Data  Collector; _ 

WORKSHEET  C1.13:  INSTRUCTOR  REACTION  INTERVIEW  FORM 


Group/Mos  //: _ _ _  Course;  _ 

Interview  Location/Environment; _ 

Interview  Starting  time; _  Instructor  ID  //(s): 


Ask  each  question  orally  along  with  the  five  possible  answers,  check  the  appropriate 
box,  and  record  the  instructor's  other  responses  in  the  space  below  each  question.  Be 
sure  he  understands  which  training  event  you  are  asking  about. 

Very  Very 

Confusing  Confusing  Undecided  Clear  Clear 

I.  Was  the  information  in  □  □  □  □  □ 

the  training  materials 
generally  clear? 

List  tasks  for  which  material  was  unclear; 


Definitely 

Not  No 

Undecided 

Yes 

Definitely 

Yes 

2.  Did  you  have  directions 
to  give  the  course? 

□ 

□ 

□ 

□ 

□ 

List  tasks  for  which  you 

needed  more  information; 

H-3S 
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WORKSHEET  C1.13  (continued) 
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Definitely 

Not 


Definitely 

No  Undecided  Yes  Yes 


3.  Was  the  information  ODD  □  □ 

in  the  material 
technically  accurate? 

List  tasks  for  which  materials  were  inaccurate: 


4.  Did  you  have  all  the 
materials  available 
for  the  course? 


About 

About 

About 

None 

i/4 

1/2 

3/4 

All 

□ 

□ 

□ 

□ 

□ 

List  tasks  for  which  you  could  not  obtain  the  materials: 


Definitely 

Not 

5.  Did  students  have  f") 

trouble  learning  any 
of  the  tasks  in  the 
course? 

List  the  tasks  the  students  had 
trouble  learning: 


Definitely 

No  Undecided  Yes  Yes 

DODD 


Give  the  reasons  you  think 
they  had  trouble  with  each  one: 


H-39 
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WORKSHEET  C1.13  (continued) 
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Additional  tasks  students  had 
trouble  learning: 


Reasons  you  think  they  had 
trouble  with  each  one: 


6.  Please  make  any  other  comments  you  would  like  to  make  about  the  course: 


H-40 


Date:  _ 

Evaluator: 
Sheet  ih 


WORKSHEET  C2:  OBSERVATION  PLAN 


Course: 

Date 

Time 

Class  # 

Type  of 
Event 

Objective  # 

Content 

Location 

Contact 

Materials 

Notes 

| 

i 

i 

Date: _ 

Evaluator: 


WORKSHEET  D3.i:  TRAINEE  CHARACTERISTICS 


Course: 


Type:  Initial  □ 
Refresher  □ 
Other:  _ 


Transition  □ 


Duty  Position: 


Date: _ 

Evaluator: 


WORKSHEET  D3.2:  INSTRUCTOR  CHARACTERISTICS 


Course: _  Times  Given: 

Code  No.: _ _  MOS: _ 


Category: 

Specified: 

Actual: 

Notes: 

Grade 

Academic 

Age 

Years  in  Army 

Training  in 
delivery  of 
training 

Experience  in 
training  delivery 

Experience  in 
subject  matter 

Involvement 
with  course 
development 

Involvement 
with  hardware 
development 

Special  aptitudes/ 
qualifications 

Method  of 
selection  to 
delivery  this 
training 

Other 

i 

•  How  instructor  was/wil!  be  evaluated? _ 

•  Pcnalties/rcwards  for  trainee  performance 


H-43 


Date:  ,  . -  .  .  - 

Evaluator: _ 

WORKSHEET  El:  SUMMARY  OF  TEST  SCORES  FOR  TASKS  OR  TEAM  FUNCTIONS 


Course:  . .  Test  Period: 

Class  //: _ 

Test  Condition/Situation: _ _ 


Type  of  Test: 


Individual  □ 
Final  □ 
Written  □ 


Crew  □ 

Within-Course  □ 
Hands-On  □ 


Code  No: _ 

Entry  Test  □ 
Pretest  *  □ 


Position: 


Individual  and/ 
or  Crew  Members 

Tasks  Selected  for  Eva] 

[nation 

a 

m 

■ 

■I 

■ 

m 

a 

a 

m 

ii 

B 

B 

B 

m 

■ 

ii 

B 

ii 

m 

gi 

■ 

■ 

B 

B 

■j 

m 

i 

m 

ca 

ii 

ii 

i 

ii 

B 

B 

B 

m 

■ 

a 

i 

B 

■i 

B 

■j 

1  1 

i 

ii 

■ 

Number  ’’No  GoM 

■ 

a 

U 

B 

B 

B 

B 

■ 

Number  of  Crews/ 
Individuals  Tested 

■ 

■ 

■ 

■ 

m 

■ 

1 

■ 

■ 

B 

B 

%  No  Go 

B 

■ 

B 

B 

a 

E 

fl 

B 

B 

H-W 


Dates _ 

Evaluator:  _ _ 

Course: _ 

Test:  _ _ : _ 

WORKSHEET  E2.1:  SUMMARY  OF  TEST  ADEQUACY  DATA  FOR  OB3ECTIVES 


Task/TLO  or  LO: 

Page _ of _ 

.for  this  objective 

PRODUCT 

PROCESS 

Questions  by  test  item: 

Questions  by  test  item: 

item  ft _ 

item  ft _ 

item  ft _ 

item  ft 

item  ft _ 

item  ft,  ,, 

item  // _ 

item  ft _ 

item  ft 

item  ft _ 

item  ft 

item  ft _ 

item  ft  - 

item  ft _ 

item  ft  __ 

item  ft 

item  ft _ 

item  ft  - 

! 

i 

Questions  by  objective  (TLO  or  LO): 


Questions  by  objective  (TLO  or  LO): 


Evaluator: 

Course: _ 

Test:  _ 


WORKSHEET  E2.2:  SUMMARY  OF  TEST  ADEQUACY  DATA  FOR  THE  TEST 


Evaluator: 

Course:  _. 

Lesson: 

WORKSHEET  E2.3:  SUMMARY  OF  PRESENTATION  DATA 

TASK/TLO  or  LOs 

PRODUCT 

PROCESS 

Questions  by  presentation  components, 

Questions  by  presentation  component, 

series  of  components,  LO,  or  task/TLO: 

series  of  components,  training  event, 

LO,  or  task/TLO; 

Questions  by  lesson  (each  one  that 

Questions  by  lesson  (each  one  that 

applies  to  this  LO  or  task/TLO,  by 

applies  to  this  LO  or  task/TLO,  by 

set  of  LOs  or  tasks/TLOs  (cf  which  this 

set  of  LOs  or  tasks/TLOs  (of  which  this 

LO  or  task/TLO  is  one),  or  by  course; 

LO  or  task/TLO  is  one),  or  by  course: 

Instructor  Code: 


Date:  _ 

Evaluator: 
Course: _ 


H-4S 


Date: _ 

Evaluator: 
Course: _ 


WORKSHEET  E3.2;  DATA  SUMMARY  BY  TRAINEE 


WORKSHEET  E3.3  DATA  SUMMARY  BY  FUNCTION 


